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ABSTRACT 
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post-high school world. The CPA's value as an assessment tool was examined in 
a pilot test that was conducted in 1996-1997 with six California high 
schools. The following three data collection activities were conducted to 
determine the CPA's effectiveness in terms of implementation, student 
performance/assessment, and whole school change: (1) 8 pilot test teachers 

and 6 employers of entry-level workers scored the CPA portfolios submitted by 
631 students from the 6 high schools; (2) 36 teachers at the 6 schools 
completed teacher surveys and participated in telephone interviews; and (3) 
623 students completed a student survey. Major findings included the 
following: (1) successful implementation of the CPA and interdisciplinary 

standards -based assessments requires multiple conditions associated with 
whole school change; (2) teachers considered the CPA a worthwhile expenditure 
of time and effort; (3) nearly two- thirds of the students indicated that they 
would recommend CPA portfolios to other students. (Nineteen tables are 
included. Appendixes, constituting approximately 40% of this document, 
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EXECUTIVE SUMMARY 



I. INTRODUCTION 

By definition, systemic reform requires schools and school systems to change a wide variety of 
practices to improve student performance. This definitional characteristic compels researchers 
to examine instruments which facilitate change across multiple dimensions of schooling (e.g., 
curriculum, grading, governance, etc.). This report begins to answer initial, broad questions 
about such an instmment, the Career Preparation Assessment portfolio (CPA) which was 
designed to bridge the theoretically interrelated reform vehicles of school-to-work (STW), 
standards-driven curriculum/assessment, and integrated curriculum. 

Overview of the CPA: The CPA is an interdisciplinary, performance-based assessment of the 
Career Preparation Standards (Thinking and Problem-Solving, Communication, Technology 
Literacy, Personal and Interpersonal, Employment Literacy, and Occupational Safety) 
designed to bolster standards-driven reform and whole school change efforts. The CPA 
defines mastery of the Career Preparation Standards (CPS) by rating portfolios as “Basic,” 
“Proficient,” or “Advanced.” These ratings also serve as indicators of hirability for an entry- 
level job. Student performance on four “dimensions” aligned with the CPS (Career 
Preparation, Analysis, Technology, and Communication) inform a portfolio’s rating. 

Questions the report addresses: Driven by data from a 1996-97 pilot test of the CPA by 
six California high schools, the report focuses on three clusters of questions: 

(1) Implementation, (2) Whole School Change, and (3) Student Performance/Assessment. 
Implementation is a critical topic because of its relation to scalability and to support for 
innovative practitioners — to make judgments about the CPA’s scalability and to allow 
practitioners to make informed decisions about the use of interdisciplinary performance- 
based assessment, we must identify the key conditions for successful implementation. 

Impact on whole school change should be understood so that the CPA’s overall value in 
the reform context can be described accurately and so that practitioners can make 
informed judgments about the need for complementary instruments or processes. The 
quality of the CPA as an assessment should be better understood to know its utility as a 
data source to guide change at the classroom, school, and district levels. 

Pilot test schools: These schools were selected from an initial pool of 300 California high 
schools most likely to be interested in and capable of successfully using the CPA. Key 
factors in selecting six pilot test schools from the 300 were: number of teachers and 
students who were to participate; teacher experience with performance-based assessment, 
working in teams, and with integrated curriculum; teacher commitment to implementation; 
and number of shared students. The six schools ranged widely in terms of geography, 
experience with performance-based assessment, and stmcture. Thirty-six teachers and 








over 600 students participated. The six schools received the following support from 
WestEd: 

• One day of implementation training 

• Two half-day trainings focused on reviewing student work against CPA requirements 
and standards 

• Guidelines about how to create a CPA portfolio for all participating students 



n. DATA SOURCES 

The data sources and methods employed were diverse, and intentionally overlapped to 
increase validity. In combination they allowed a thorough analysis of the three clusters of 
questions. The following provided important data for analysis: 

Student portfolios . Students from the six pilot test schools submitted CPA portfolios. 
Eight pilot test teachers and six employers of entry-level workers scored the portfolios as 
Basic, Proficient, or Advanced over a two-day period in July, 1997. Complete portfolios 
were also given a score on each of four dimensions (Career Preparation, Analysis, 
Technology, and Communication). Each complete portfolio was scored by four different 
scorers. WestEd staff performed statistical analyses of the scoring of the portfolios 
regarding overall reliability, reliability among types of scorers (teachers and employers), 
the impact of dimensional scores on overall scores, and student performance by ethnicity 
and gender. 

Scorer Evaluations . After the portfolios were scored, each scorer responded to a one- 
page questionnaire about scoring issues and the CPA as an indicator of “hirability.” 

WestEd Observation of Scoring Session . WestEd staff took note of key developments 
during the scoring session, i.e., which dimensions raised the most questions during training 
and calibration. 

Teacher Survey . At the end of the school year, pilot test teachers completed surveys 
about their experiences with the CPS/CPA. Survey results were analyzed as a group and a 
comparison of survey results between higher-performing and lower-performing schools 
was conducted. 

Teacher Interviews . Telephone interviews were conducted with the lead teacher from 
each pilot school to obtain more detailed information about the issues studied in the 
teacher survey. 

Student Survey . Students who participated in the pilot testing completed a survey that 
gathered information about their perceptions of the CPS/CPA, including their value to 
employers and colleges and how the CPS/CPA was implemented. Survey results were 
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analyzed as a group and a comparison of survey results between higher-performing and 
lower-performing schools was conducted. 



Evaluations of Professional Development . Teachers completed a brief feedback form at 
the end of each CPA professional development workshop they attended. 



m. FINDINGS 

Implementation 

Key conditions for successful implementation of the CPA: “Successful” implementation 
of the CPA is defined by the following two characteristics. First, at least 40% of a 
school’s students who attempted a CPA portfolio actually completed the required entries. 
Second, a majority of the completed portfolios received a score above the Basic rating. 
Three of the six pilot test schools met the two criteria for success. 

Teacher surveys and interviews, student surveys, and students’ portfolios informed the 
identification of the following key conditions: 

• teacher experience with the CPA and/or portfolios; 

• integrated curriculum; 

• regular and effective methods of communication by implementing teachers; 

• shared students; 

• supporting professional development; 

• initiation of implementation of the CPA early in the school year; and, perhaps most 
importantly, 

• teacher commitment to implementation of the CPA and to students attaining the CPS. 

It is worth noting that two of the three higher-performing schools had all of the essential 
conditions with the third having all but one (experience with the CPA and/or portfolios). 
This suggests that successful implementation of the CPA and of interdisciplinary 
standards-based assessments requires multiple conditions associated with whole school 
change. 

Teacher and student views toward the CPA and CPA : Teachers viewed the CPA as a 
worthwhile expenditure of time and effort. Teachers in the pilot study provided a strong 



endorsement of the CPA when 92% of them “agreed” with the statement “I intend to 
continue using the CPA next year.” Other teacher survey results include: 

♦ The CPA portfolio is important for my students (94% agreeing). 

♦ The CPS represent skills valued by employers (89% agreeing). 

♦ The CPS represent skills valued by post-secondary educational institutions (83% 
agreeing). 

♦ The CPA assesses the CPS well (81% agreeing). 

♦ Doing the CPA is a good way for students to learn or master the CPS (86% agreeing). 



Students generally found the CPA to be valuable. Nearly two-thirds of students indicated 
that they would recommend CPA portfolios to other students. Interestingly, their 
responses indicate that they saw both the CPA and CPS as having some value, with 
mastery of the CPS perceived as more important than actually doing a CPA portfolio. 
Students at higher-performing schools were consistently more positive about the 
CPS/CPA than students at lower-performing schools. Furthermore, survey data suggest 
that CPA implementation at lower-performing schools needs more teacher time and more 
focus on the CPS. Less than half of students at lower-performing schools reported that 
they received the information and help needed to be successful with the CPA, or that their 
teachers spent enough time explaining the CPS. Given that teachers involved in the pilot 
test received the same professional development and materials, these data suggest that 
teacher impact on student support for the CPA is substantial. 

Whole school change 

The CPA and whole school change : Forty-four percent of teachers agreed that “Our 
school or program has changed its structure, procedures, or curriculum based on CPA- 
related student work.” However, the depth to which the CPA caused implementation of 
components of whole school change is not known— pilot test schools were implementing 
components of whole school change prior to the pilot test. Importantly, teachers were 
positive about the CPA’s capacity as a vehicle to affect a key component of whole school 
change, standards-based curriculum and assessment — 83% of pilot test teachers agreed 
that “The CPA is, or could be, a valuable support for implementing standards-based 
curriculum and assessment schoolwide.” It is important to note that when identifying 
factors they believe supported successful CPA implementation, teachers often identified 
conditions commonly associated as components of whole school change. This may lead to 
teachers working to create those conditions as they attempt to improve CPA 
implementation. 

Changing classroom practice : Responses from the teacher survey suggested the CPS and 
CPA had impacts on teaching practice. Fifty-nine percent of teachers reported that “Using 
the CPS made me alter or rethink my teaching methods” and that “Using the CPA 
portfolio as a means of organizing student work made me alter or rethink my teaching 
methods.” Regarding students’ perception of teachers who use the CPA and those who 
do not, it is interesting to note that 44% of students “agreed” that “Teachers who use the 
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CPA teach differently or ask me to complete different kinds of assignments than teachers 
who do not use the CPA.” 

Student performance/assessment 

Background : WestEd received 631 student portfolios from the pilot schools. Of these 
portfolios, only 20% were “complete,” meaning the portfolios contained required entries. 
The percentage of completed portfolios received from individual schools ranged from 98% 
to 0%. Three levels of performance (Basic, Proficient, Advanced) were used when giving 
an overall score. Student performance on the four dimensions aligned with the CPS 
informed the overall rating of a portfolio. Here are definitions for the performance levels: 

Basic: This rating is given to work that needs improvement and is not ready to 
show an employer. While the work may be complete, it does not show mastery of 
the CPS and does not indicate that the student has the skills necessary for 
satisfactory performance in an entry-level position. 

Proficient: This rating means the work is good and is ready to show an employer. 
The work is complete and meets all requirements; it shows mastery of the CPS and 
indicates that the student has the skills necessary for satisfactory performance in an 
entry-level position. 

Advanced: This rating means that the work is outstanding. It may exceed the 
expectations of an employer. The work is complete and meets all requirements; it 
shows strong mastery of the CPS and indicates that the student may have the skills 
necessary for exemplary performance in an entry-level position. 

Student performance : Sixty-two percent of students submitting complete portfolios 
received Proficient or Advanced scores. Nearly one-fourth of the portfolios (23%) were 
scored Basic. The remaining portfolios (15%) fell between Basic and Proficient. These 
percentages are not representative of the quality of the submitted incomplete portfolios 
which, while not formally scored, generally were of much lower quality than the complete 
portfolios. 

Substantial differences in student performance existed among the schools and within the 
three higher-performing schools. Of the three higher-performing schools, Kennedy had 
the highest percentage of student portfolios scored as Proficient or Advanced (83%) — 
significantly greater than comparable results at Washington (61%) or Keith (63%). 

Student performance varied widely by gender, race, and school. 

Given that this is the first relatively broad pilot test for the CPA, it is difficult to state 
whether student performance was “good,” “bad,” or “mediocre.” The performance of the 
three higher-performing schools’ (Washington, an urban school; Kennedy, suburban; and 
Keith, rural) students suggests strongly that the CPA is implementable — substantial 
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numbers of ethnically diverse students can complete CPA portfolios to a Proflcient, 
or potentially ‘‘hirable,” level. 



Reliability : The reliability of scores given to CPA portfolios was examined in two ways. 
First, portfolios were analyzed to identify how many of the four scorers agreed on the 
overall rating. Second, an analysis was conducted to determine the degree to which there 
was consistency among pairs of scorers (educator-educator, educator-employer and 
employer-employer). 

Given that this is the first attempt at scoring substantial numbers of portfolios from a 
diverse group of schools, the reliability of the CPA was promising. Seventy-five percent 
of the time, at least three of the four scorers gave the same overall score to a particular 
portfolio. With pairs of scorers reliability was lower, 64% overall. Teachers, compared to 
employers, rated a greater percentage of portfolios as Proficient and fewer as Basic. 



IV. CONCLUSION 



The pilot test shed light on the CPA’s potential as a scalable, reliable assessment that 
supports systemic reform and improved student performance. We now know that the 
CPA can be implemented, at least by a range of diverse schools possessing certain 
preconditions, such that student performance exceeds expectations informed by six years 
of experience with standards-based portfolios (62% of completed portfolios were 
Proficient or better). Furthermore, pilot test teachers believe the CPA is a valuable 
teaching tool that can promote changes in classroom practice and schoolwide reform. 
Conversely, it is clear that successful implementation is in no way assured even when 
implementing schools are stronger in terms of capacity for innovation than the general 
school population — the three lower-performing CPA schools went through a meaningful 
recruiting and selection process aimed at identifying schools well-positioned to use the 
CPA. In spite of this, student performance for these three schools suggested that the CPA 
may not be implemented well even under relatively positive circumstances. We also now 
know that CPA portfolios can be scored with a reasonable level of reliability by a diverse 
group of scorers. Moreover, we have data about scoring that will almost certainly lead to 
improved reUabUity. Overall, the CPA has a solid base from which to progress. 

For the CPA to realize its potential and to gain greater understanding of its scalability, it 
must meet interrelated challenges regarding: (1) its reliability as an assessment; 

(2) modification of the portfolio’s entries and scoring rubric to better address the needs of 
employers and college admissions processes; and (3) implementation in a larger number of 
schools representative of the general population of schools with relatively high stakes. 
Regarding reliability and validity, while the CPA needs improvement, it is off to a good 
start technically. Now, the CPA needs to achieve a level of reliability and validity which 
allows educators, employers, and parents in selected high school communities to use it as 
a meaningful indicator of student performance. The following steps should increase 
reliability: revise the scoring rubric with learnings from the 1996-97 pilot test scoring in 
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order to better define the three performance levels and dimensions, revise scorer training 
procedures aimed at building a conunon definition of the three performance levels, and 
continue to monitor and analyze scoring events and results. After taking these steps, 
reliability should equal or exceed that of the few other available performance-based 
assessments measuring work readiness skills. 

To increase its value as an assessment and its scalability, the CPA must better meet the 
needs of employers and college admissions officers. Regarding employers, the Proficient 
rating must better represent “hirable” in an entry-level position. Regarding college 
admissions, while the CPS align with skills colleges want in their students, formal review 
and validation of the CPS by post-secondary institutions is needed. Review of the CPA’s 
format is also necessary to see if post-secondary educators’ concerns about use of 
portfolios in admissions processes can be addressed. 

Implementation in schools more representative of the general population than the pilot test 
schools, and with relatively high stakes (e.g., determining the grade for two or three 
courses or serving as a high school graduation requirement), is essential to better 
understand the CPA’s scalability and impact on whole school change. Implementing the 
CPA in such a fashion would allow deeper understanding of the necessary preconditions 
for successful implementation, particularly with regard to implementation by teachers less 
interested in the CPA than those in the pilot test. It would also allow for increased study 
of external and consequential validity. High stakes implementation of the CPA in schools 
more representative of the general population than the pilot test schools will require an 
aggressive recruiting effort and may require modification of the CPA and CPS to meet 
local needs. Willingness to tailor the portfolio components and standards to local needs 
should increase the following: the pool of schools interested in using the CPA as a high 
stakes assessment, educator desire to use performance on the CPA as meaningful data 
informing systemic change, employer desire to use the CPA in hiring decisions, and 
admissions officers’ desire to use it in admissions decisions. 

If the CPA meets the challenges listed above, and if preconditions necessary for successful 
implementation come to exist in a large number of schools, the CPA could become a 
scalable tool that bolsters standards-driven curriculum, performance-based assessment, 
and whole school change. 
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INTRODUCTION 



By definition, systemic reform requires schools and school systems to change a wide variety 
of practices in order to improve student performance. This definitional characteristic 
compels researchers to examine instruments which facilitate change across multiple 
dimensions of schooling (e.g., curriculum, grading, governance, etc.). This report begins to 
answer initial, broad questions about such an instrument, the Career Preparation Assessment 
portfolio (CPA), an interdisciplinary, performance-based assessment of skills essential for 
the post-high school world. The CPA was designed to serve as such an instrument by 
providing a bridge between the theoretically interrelated reform vehicles of school-to-work 
(STW), standards-driven curriculum/assessment, and integrated curriculum. It derives much 
of its potential as a support for systemic reform from the large investment in these three 
vehicles by schools and reformers attempting whole school change — the CPA may add 
coherence to many local reform efforts. 

The report is organized by broad questions grouped into three clusters: 

(1) Implementation . (2) Student Performance/Assessment , and (3) Whole School Change . 
Implementation is a critical topic because practitioners considering interdisciplinary 
standards-based curriculum and assessment must know more about what it takes to 
actually do them. Also, to make judgments about the CPA’s scalability, we must identify 
the key conditions for successful implementation of the CPA as well as teacher and 
student views about the utility of the CPA. The quality of the CPA as an assessment 
should be understood in order to know its utility as a data source to inform change at the 
classroom, school, and district levels. Furthermore, analyzing student performance on 
the CPA will tell much about its utility and implementability as an assessment and as a 
curricular tool. Actual impact on whole school change should be understood so that the 
CPA’s overall value in the reform context can be described accurately and so that 
practitioners can make Judgments about the need for complimentary instruments or 
procedures. 

Data from a pilot test of the CPA conducted during the 1996-97 school year with six 
California high schools provide the basis for preliminary answers to the questions. These 
answers should inform the design and use of the CPA such that in the 1998-99 school 
year the CPA could undergo a final test and redesign period leading to the completion of 
the development of the CPA as a reliable, scalable performance-based assessment of 
skills essential for the post-high school world that also actively supports whole school 
change and systemic reform. 

Study of, and support for, the CPA aligns closely with WestEd’s approach to its national 
specialty area of Assessment — improving student performance and bolstering whole 
school change through focus on four priority areas: whole school implementation, 
school-to-work transition, teacher assessment and certification, and development and use 
of scoring rubrics. 
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OVERVIEW OF THE CAREER PREPARATION ASSESSMENT PORTFOLIO 



The CPA is an interdisciplinary, performance-based assessment of the Career Preparation 
Standards, skills essential for the post-high school world. The Career Preparation 
Standards (Thinking and Problem-Solving, Communication, Technology Literacy, 
Personal and Interpersonal, Employment Literacy, and Occupational Safety) align with 
other important STW skills and standards frameworks such as those in the SCANS 
report. (The Career Preparation Standards may be found in Appendix A.) Schools and 
districts can use these standards as part of their systemic reform programs, such as Goals 
2000, Carl Perkins, and the School-To-Work Opportunities Act. 



The CPA has multiple purposes. The first is to foster and assess student achievement of 
the Career Preparation Standards (CPS). Second, it is designed to bolster standards- 
driven reform efforts that focus on preparing students for the post-high school world. 
Schools, particularly those emphasizing school-to-work transition, can align teaching, 
learning, and assessment to support student achievement of the CPS. Third, and related 
to the second purpose, the CPA provides a foundation for teacher collaboration and 
interdisciplinary curriculum. Because students can demonstrate one or more of the CPS 
in any class, teachers from all disciplines can support the CPA’s use. Fourth, the CPA 
gives students a useful tool in applying for either entry-level employment or admission to 
post-secondary education: a completed CPA portfolio. 

The CPA portfolio, developed by WestEd in collaboration with teachers and employers, 
is a set of cumulative entries compiled by students during one or more semesters. The 
portfolio entries are flexible and can be completed in academic and vocational education 
classes. The entries are presented in Table 1. When scored against the CPA rubric, 
complete portfolios receive an overall rating of Basic, Proficient or Advanced. A 
“Basic” rating should serve as an indicator that the student has not mastered the CPS, is 
not hirable for an entry-level job in a non-technical field, and that the portfolio is not 
ready to show to an entry-level employer. The “Proficient” rating is an indicator that the 
student has mastered the CPS, is hirable for an entry-level job in a non-technical field, 
and that the portfolio is ready to show to an entry-level employer. The “Advanced” 
rating is an indicator that the student has strong mastery of the CPS, is hirable for an 
entry-level job in a non-technical field, and that the portfolio may exceed the 
expectations of an entry-level employer. It should be noted that the ratings are also 
designed to describe the quality of the portfolio relative to the expectations of college 
admissions officers. However, further research is necessary to understand the portfolio’s 
use in the admissions process. 

Student performance on four “dimensions”, or sub-categories within the rubric, inform 
the overall rating of a portfolio. Each dimension (Career Preparation, Analysis, 
Technology, and Communication) contain descriptors of student performance against one 
or two of the seven CPS at the Basic, Proficient, and Advanced levels. For example, a 
descriptor of written communication at the Advanced level is “Writing is clear and well 
organized throughout the portfolio.” The analogous descriptor at the Basic level is “Ideas 
are presented in a disorganized way.” Boiling down the seven CPS into four essential 
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dimensions appeared necessary for efficient scoring — a rubric with seven dimensions is 
quite hard to score in a reasonable period of time and was less supportive of holistic 
scoring than a rubric with four dimensions. 

Table 1 



CPA portfolio entries 



Entry 


Description 


Personal statement 


Students outline their career goals and evaluate their skills in relation to the 
CPS. 


Resume 


Students prepare a one-page resume describing their experiences and skills in 
relation to the CPS. 


Application 


Students obtain and complete an application for employment or continued 
education/training. 


Letter of 
recommendation 


Students obtain a letter of recommendation from someone who knows them 
well, such as a supervisor, community leader, or teacher. 


Work samples 


Students select pieces of work demonstrating their mastery of the CPS. 
Work samples can range from a science experiment to organizing a 
community event to a statistical analysis of a school-wide survey. (One of 
the two work samples must address Technology Literacy). 


Writing sample 


Students select any document demonstrating their ability to communicate 
in writing and to reach a conclusion based on supporting information and 
evidence. Students are evaluated on their writing ability and analytical 
reasoning. Writing samples can range from a comparative analysis of 
short stories to a business proposal. 


Interpersonal skills 
evaluation 


An evaluation of students’ interpersonal skills (team work, leadership, 
etc.), the ISE is completed by a supervisor or teacher after a work 
experience, a team project, or a class. 


Optional entries 


Students may create an additional section in their portfolios and include 
any of the following: 1) attendance records, 2) transcript with GPA, 3) 
extra-curricular activities, certificates, and awards, or 4) cover letter. The 
CPA rubric does not address optional entries. 



In practice, standards-driven curriculum, assessments, and school-to-work co-exist at a 
small number of high schools, sometimes complementing whole school change, 
sometimes working independently from or at odds with whole school change. While 
many educators see the need for whole school change and for supporting standards- 
driven curriculum and assessments in a STW context, few possess the formidable 



O 

ERIC 



3 



14 



combination of will, resources, and expertise to implement. For those who are willing to 
implement comprehensively, the CPA appears to help blend standards-driven curriculum, 
assessments, and school-to-work transition, which may lead to whole school change. The 
CPA is unique as a tool that, in the context of STW, is relatively easy to implement and 
coalesces curriculum, standards, and assessment across multiple disciplines at the high 
school level. 

To date, the CPA’s “blending” ability and its value as a teaching tool have been the basis 
for teachers choosing to implement it. In particular, the CPA has served as a relatively 
efficient method to organize teaching and student work around the CPS. The word 
“organize” is chosen carefully. Over three years of use in the field, what appears to have 
attracted teachers to actually use the CPA is its provision of a framework for eliciting and 
organizing student performance toward the CPS. In practice it has served not as 
complete curriculum or assessment, but as a mixture of the two. For example, at Keith 
High School' in rural northern California, teachers of English, Agriculture, and Business 
have their students complete the CPA portfolio, creating portfolio entries in all three 
classes. In none of these three classes does the CPA serve as the “core” curriculum. 
However, each teacher makes sure that his students understand the CPS and identify their 
best work relative to those standards. Furthermore, while the portfolio comprises a major 
portion of students’ final grades, teachers do not yet use the CPA as an assessment of 
individual CPS. And according to the teachers, CPA portfolios are the most important 
single product their students create during high school — the portfolios are the place 
where students best demonstrate essential skills and their individual personalities. 



HISTORY 

1994-95 

Development of the CPA began in the winter of 1994 when California’s legislature 
passed AB 198 which mandated schools to report on students’ readiness for the world of 
work. The California Department of Education (CDE) then contracted with WestEd to 
develop an assessment which: supported high schools’ compliance with the new law, 
aligned with the State’s new high school reform vision (Second to None), and improved 
student attainment of California’s equivalent of the SCANS skills and competencies, the 
Career Preparation Standards. The new assessment was to build on and align with 
WestEd’ s pioneering work with the CDE in the field of performance-based assessment of 
occupational cluster standards, the Career-Technical Assessment Program (C-TAP). 

During the winter of 1994 WestEd staff, in collaboration with educators and employers, 
designed the entries that comprise the CPA portfolio and the “Student Guidelines,” which 
assist students and teachers in completing the entries. WestEd then recruited eight 



‘ Pilot test schools are described using pseudonyms. 
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diverse schools to pilot test portfolio entries (not the complete portfolio) during the 
spring of 1995. Nevertheless, students from two of the eight pilot sites submitted 
complete portfolios. WestEd staff, pilot test teachers, and employers with experience in 
entry-level hiring evaluated the 2,000+ completed entries, creating exemplar portfolios 
for the Proficient and Advanced levels for use in teacher training and with students. 

In addition to analyzing entries, the following key activities took place: 

• WestEd developed and analyzed student and teacher surveys regarding the utility and 
impact of the CPA. Major findings of the 1994-95 surveys were that students and 
teachers found the CPA easy to use, the CPA is a powerful teaching and learning tool, 
and that many students saw a clear tie between success with the CPA and 
preparedness for the world of work. 

• The “Student Guidelines” for implementing the CPA were revised using data from 
the pilot and feedback from pilot teachers. 

• A scoring rubric, aligned tightly with the student work received in the spring pilot, 
was drafted and refined after review by practitioners and employers. 

• Dissemination work began aggressively, with presentations at major assessment and 
systemic reform conferences and a mailing to some 200 targeted high schools. 

From our work in 1994-95, we learned the CPA had strong potential as an implementable 
comprehensive assessment that organically promotes curriculum integration and teacher 
collaboration. Also, the work generated initial data about necessary preconditions for 
successful implementation - schools with supportive structures (integrated curriculum, 
team teaching, common planning time) and prior experience with performance-based 
assessment submitted more and higher quality student work. 

1995 - 96 

In the 1995-96 school year, state support for the CPA was withdrawn following a major 
restructuring of the CDE. However, the two schools that submitted complete portfolios 
in 1994-95 and one other school from the first year of use continued to pilot test the CPA 
with support from WestEd. Some 150 completed portfolios were submitted by the three 
schools. With support from OERI the portfolios were evaluated, exemplars selected, and 
modifications to the guidelines, scoring rubric and professional development were made 
based on teacher and student surveys and other feedback. Also, teachers from some of 
these pilot test schools provided substantive input into the design of the 1996-97 pilot 
test, adding new information about key conditions for successful completion of CPA 
portfolio. 

1996 - 97 

The 1996 year was pivotal for the CPA. Limited pilot testing in 1994 and 1995 had been 
successful, and informal data-gathering suggested that high schools across California 
were at a beginning readiness level to try a performance-based assessment of generic 
work readiness skills. WestEd decided to launch a significant test of the CPA across 



several factors: usability, reliability as an assessment, and impact of use on whole school 
change. 

The 1996-97 pilot testing of the CPA was designed to build on development work 
completed during the 1994-95 and 1995-96 school years. To start, WestEd identified the 
300 California high schools most likely to have interest in, and capacity for, 
implementing the CPA and invited them to apply for participation in the 1996-97 pilot 
test. Twenty-eight schools submitted four-page applications. From these 28, WestEd 
selected seven that appeared able to implement well and represented diversity of 
geography and ethnicity. Key factors in selection were; number of teachers and students 
who were to participate; teacher experience with performance-based assessment, working 
in teams, and with integrated curriculum; teacher commitment to implementation; and 
number of shared students. The quid pro quo for participation in this larger pilot test was 
that WestEd would provide materials and professional development, while participating 
teachers and students would submit their portfolios and observations about implementing 
the CPA to WestEd for analysis. 

As Table 2 details, the selected schools range widely in terms of geography, experience 
with performance-based assessment, and structure. With the exception of Keith, all 
schools were ethnically diverse. It must be noted that while academies functioning as 
comprehensive schools-within-schools participated in the pilot test, only Keith required 
all students in one grade (seniors) to complete a portfolio. All schools accepted into the 
pilot were judged to have the capacity to have their students produce substantial numbers 
of complete portfolios — no schools were accepted that appeared to lack the 
fundamentals necessary for completed portfolios. Six schools (one of the original seven 
schools dropped out prior to the beginning of the 1996-97 school year), 36 teachers, and 
over 600 students participated. The six schools received professional development about 
the CPA from WestEd. 
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Table 2 



Descriptive information about schools participating in the 1996-97 CPA pilot test 



School 


# OF Students 
Projected to 
Use the CPA 


Geographic 

Location 


Prior Experience 
WITH Performance- 
Based Assessment 


School Structure 


Tunnel 


90 


(suburban) Bay Area 


low 


informal academy 


Keith 


40 


(rural) northern CA 


high 


traditional 


Roosevelt 


350 


(urban) Los Angeles 


low 


formal academy 


Washington 


80 


(urban) Pasadena 


high 


formal academy 


Lincoln 


140 


(suburban) Santa Cruz 


low 


continuation 


Kennedy 


125 


(suburban) Bay Area 


low 


formal academy 



Pilot test schools achieved widely varying results, which will be detailed in the 
assessment and implementation sections. 
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METHODOLOGY 



DATA SOURCES AND INSTRUMENTS 

Throughout the 1996-97 school year, and into the following summer, WestEd gathered 
data to help understand the CPA in terms of: (1) Implementation . (2) Student 
Performance/Assessment , and (3) Whole School Change . A list of data sources follows. 

Student portfolios. Students from six schools submitted CPA portfolios. Eight pilot test 
teachers and six employers of entry-level workers scored the portfolios over a two-day 
period in July, 1997. All complete portfolios (those containing required entries) were 
given a score on each of four dimensions (Career Preparation, Analysis, Technology, and 
Communication) and an overall score. Furthermore, each complete portfolio was scored 
by four different scorers. WestEd’ s approach to scoring portfolios during this pilot test 
was more extensive than the scoring procedures generally used with portfolios. In most 
portfolio systems, only one scorer scores each portfolio. A second scorer then scores a 
sample of the first scorer’s portfolios to be sure that the first scorer applied the scoring 
rubric correctly. If the second scoring shows the first scorer is applying the rubric 
properly, it is assumed that the first scorer scored all the portfolios correctly. Scoring 
each portfolio rnultiple times generated rich data which WestEd used to explore issues 
related to the reliability with which CPA portfolios could be scored, how dimension 
scores related to the overall scores given to portfolios, and for biases among different 
types of scorers (e.g., educators and employers). 

Teacher Survey. At the end of the school year, pilot test teachers were surveyed about 
their experiences with the CPS/CPA. The survey focused on the following areas: 

♦ the extent to which the CPA was implemented in the teacher’s classrooms; 

♦ overall impressions of the CPA; 

♦ the degree to which using the CPS/CPA affected teaching methods or curriculum; 

♦ the most important factors contributing to the successful implementation of the CPA; 

♦ resources which would have made CPA implementation more successful; 

♦ professional development for the CPA; and 

♦ education reform efforts occurring at the school, excluding the CPA, such as work 
with standards, integrated curriculum, block scheduling, team teaching, and career 
academies. 

Teachers responded to most of the items on a Likert-type scale which asked how strongly 
they agreed or disagreed with a statement. Thirty-six surveys were completed by 
teachers at the six pilot schools. 

Teacher Interviews. Telephone interviews were conducted with the lead teacher from 
each of the pilot schools to obtain more detailed information about the issues studied in 
the teacher survey. 
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Student Survey. Students who participated in the pilot testing completed a survey that 
gathered information about the following areas: 

♦ their perceptions of the CPS/CPA, including their perception value of the CPA to 
employers and colleges; and 

♦ how the CPS/CPA were implemented in their classrooms. Issues included whether 
they understood the CPS and the CPA, the level of assistance they received from their 
teachers, and how classrooms where the CPA was used differed from other 
classrooms. 

Most of the items were on a Likert scale which asked how strongly they agreed or 
disagreed with a particular statement. Some items also asked students to explain their 
responses. A total of 623 students completed the survey. 

Scorer Evaluations. After the portfolios had been scored, each scorer responded to a one- 
page questionnaire asking whether the dimensional scoring helped or hindered them in 
arriving at a portfolio’s overall score, which dimensions most influenced their scoring, 
whether the rubric should have three or four score points, and whether they felt 
“Proficient” was a good indicator of “hirability.” Six employers and eight educators 
completed the scorer evaluation. Each of the educators came from a pilot school. 

Evaluations of Professional Development. Teachers were asked to complete a brief 
feedback form at the end of each CPA professional development workshop they attended. 
The feedback forms contained several Likert scale items about whether training 
objectives were met and how useful teachers found the information presented during the 
workshops. Space was provided so teachers could write comments about what was most 
helpful about the training session. 

A copy of each data collection instrument may be found in Appendix B. 
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FINDINGS 



This section of the report is organized according to the clusters of questions listed in the 
Introduction (Implementation, Whole School Change, and Student 
Performance/Assessment). Each of the three clusters is organized by key questions or 
sub-topics, in which results, analyses, and “Summary of Findings” are presented. 

I. IMPLEMENTATION 

This section of the report seeks to answer the following questions: “Did teachers receive 
appropriate support to implement the CPA?”, “Did teachers view the CPA as a 
worthwhile expenditure of time and effort?”, “Did students value the CPA?”, “Did 
students value the CPS?”, and “What are the key conditions for successful 
implementation of the CPA?” 



DID TEACHERS RECEIVE APPROPRIATE SUPPORT TO IMPLEMENT THE CPA ? 

The answer to this question has deep implications for taking the CPA to scale — if the 
relatively limited support WestEd provided for pilot schools was sufficient, going to 
scale might be relatively inexpensive. While it is early in the development and study of 
the CPA to draw definitive conclusions about this question, teacher surveys, teacher 
interviews, and the quality of the portfolios produced provide meaningful data for initial 
analysis. 

With one exception (Keith HS), each school received the following support: 

• one day of implementation training; 

• two half-day trainings, focused on reviewing draft entries against CPA requirements 
and standards; and 

• student guidelines for all participating students. 

Professional development : WestEd structured initial professional development during 
the pilot test to help teachers understand the CPA and how to implement it in their 
classrooms. Follow-up professional development provided opportunities for teachers to 
review their students’ work against CPA requirements and the CPS, and to further 
explore how they could integrate the CP’A into their classrooms. 

Responses from feedback forms teachers completed at the end of each professional 
development session (teachers from each school attended one session in the Fall and two 
in the Spring) showed that teachers gave high marks to the professional development 
they received from WestEd. Teachers consistently reported that workshops did an 
“Excellent” or “Good” job providing information about the following: the CPA 
portfolio, tasks that elicit work demonstrating the CPS, evaluating student work using the 



CPA rubric, and ways to support the development of the most challenging portfolio entry, 
work samples. In their open-ended comments, teachers reported that reviewing sample 
portfolios and, in later sessions, reviewing samples of work from each other’s classrooms 
were particularly useful activities. No item on the feedback form consistently received a 
response worse than “Good.” 

'y 

The responses to questions about professional development in the teacher survey 
completed at the end of the school year , after teachers had used the CPA in their 
classrooms, were not uniformly positive. Although teachers reported that they 
understood most aspects of the CPS/CPA, there were specific areas related to 
implementing the CPS/CPA where teachers indicated more professional development 
would be useful. These areas were primarily related to integrating the CPA into teachers’ 
curricula and developing student assignments. 

There were distinct differences in how much teachers felt they understood various 
aspects of the CPS/CPA. Furthermore, teachers indicated that their theoretical grasp of 
the CPA was greater than their understanding of how to implement it in the classroom. 
Between 80% and 90% of teachers “agreed”^ that they understood the purposes of the 
CPA, the components of the CPA, and the CPS. About two-thirds of the teachers 
believed that they understood how to judge when student work demonstrates the CPS, the 
nature of the tasks that elicit work demonstrating the CPS, and how to use the CPA 
rubric. Teachers were much less likely to understand how to integrate the CPA into their 
curriculum plans (42% of teachers “agreed”) and how to design assignments which help 
students demonstrate the CPS (39% agreeing). These data suggest that additional 
training in the latter two areas might be needed, particularly since they deal with 
irriportant implementation issues. Table 3 presents information from the teacher survey 
about the professional development that teachers received. 

Table 3 

Percent of teachers who “Strongly Agreed” or “Agreed” that they understood specific 
aspects of the CPS/CPA after attending CPA professional development workshops 



Aspect of CPS/CPA 


Percent Agreeing 


The purposes of the CPA portfolio 


89% 


The Career Preparation Standards 


83% 


The components of the CPA 


83% 


How to judge when student work demonstrates the CPS 


69% 


How to use the CPA rubric 


67% 


The nature of tasks that elicit work demonstrating the CPS 


66% 


How to integrate the CPA into my curriculum 


42% 


How to design assignments which help students demonstrate the CPS 


39% 



^ Complete results from the CPA teacher survey and the CPA Student Survey are presented in Appendix C. 
^ A teacher was considered to “agree” with a survey item if he/she marked “Strongly Agree” or “Agree” on 
a Likert-type item. 
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In interviews with lead teachers and informal conversations with other CPA teachers 
aimed at understanding why three of the six pilot test schools did not submit higher 
numbers of complete or above-Basic portfolios provided additional data about 
professional development. During these interviews, professional development was never 
cited as a potentially causal factor in increased overall success. 

Student Guidelines: Teachers found the Student Guidelines to be valuable, with 36% 
identifying the Guidelines as one of the five most important factors in the success of their 
CPA efforts. Furthermore, during professional development sessions, teachers regularly 
commented on the high quality of the Guidelines. Only 6% of teachers identified 
needing different guidelines as being support they wanted but lacked. 

Portfolio completion rates and quality: As is detailed in the Student 
Performance/Assessment section of this report, the quality and completion rates of 
portfolios varied widely by school. However, three of the schools had completion rates 
ranging from just under 50% to 98% with large majorities of completed portfolios above 
the Basic rating. The other three schools had quite low completion rates (0-12%) that 
suggest that CPA implementation at those schools needs substantial improvement. 

Summary of Findings: Given that half of the pilot test schools produced meaningful 
numbers of complete, above-Basic portfolios it is reasonable to posit that, for these three 
relatively diverse schools, support was adequate, and that while not necessarily being 
causal in their success, was valuable. While more and/or better professional development 
might be an essential ingredient for improved performance, teachers consistently 
reported that it would not have been causal in increased success. Furthermore, it is 
unlikely that professional development could significantly compensate for a variety of 
other structural factors (i.e., shared students, common planning time, commitment to 
successful implementation, and experience with the CPA and performance-based 
assessment). 



DID TEACHERS VIEW THE CPA AS A WORTHWHILE EXPENDITURE OF TIME 
AND EFFORT? 

Like the prior question, this one relates to scalability — if pilot test teachers, who, by 
their very nature have shown strong interest in the CPA, do not answer “Yes” to this 
question, getting other teachers to successfully implement the CPA faces immense 
challenges. 
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Overall, teachers had very positive attitudes toward the CPS/CPA. An analysis of Likert- 
type Items from the teacher survey indicated that over 80% of teachers “agreed” with the 
following statements: 

♦ The CPA portfolio is important for my students (94% agreeing). 

♦ The CPS represent skills valued by employers (89% agreeing). 

♦ The CPS represent skills valued by post-secondary educational institutions (83% 
agreeing). 

♦ The CPA assesses the CPS well (81% agreeing). 

♦ Doing the CPA portfolio is a good way for students to learn or master the CPS (86% 
agreeing). 

Additionally, between 70% and 80% of teachers “agreed” that: 

♦ CPA portfolios are of interest to employers (78% agreeing). 

♦ Using the CPA improves students’ academic work (75% agreeing). 

♦ The CPA is a valuable assessment tool for me (75% agreeing). 

♦ My students were motivated and interested in participating in the CPA (72% agreeing). 

Together, these responses provide strong evidence that teachers believe the CPS and 
CPA portfolios are valuable, focusing on skills that are important for students’ futures 
and improving the quality of student work. Finally, and perhaps most importantly, 
teachers in the pilot study provided a strong endorsement of the CPA when 92% of them 
“agreed” with the statement “I intend to continue using the CPA next year.” 

Since there were distinct differences among schools in the percent of completed 
portfolios students produced, teachers’ attitudes were examined to see whether they 
related to portfolio completion rates. An analysis comparing the attitudes of teachers 
from high completion rate schools (44% to 98% of portfolios completed) to those from 
low completion rate schools (0% to 10% completed) showed no consistent differences in 
attitudes. 

S ummar y of Findings: Given that implementing the CPA or any substantial 
performance-based assessment is difficult and that many teachers did not produce 
significant numbers of complete or above-Basic portfolios, it is remarkable that 92% of 
teachers plan to use the CPA in the 1997-98 school year. Teachers believe that the CPA 
is a worthwhile expenditure of time and effort. 



DID STUDENTS VALUE THE CPA ? 

This question is the important student analog of the previous question for teachers. 
While results of any survey of teenagers should be viewed with some skepticism, the 
results of the CPA survey provide insight into the answer to the question. 
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Surveys were completed by the students at the pilot high schools who worked with the 
CPS/CPA during the 1996-97 school year. These surveys focused on two topics, their 
attitudes toward the CPS/CPA and their experiences implementing the CPA. Like their 
teachers, students reacted positively to the CPS/CPA. Nearly two-thirds of students 
indicated that they would recommend CPA portfolios to other students. Tables 4 
and 5 present the student survey data in full. 

Regarding how students’ attitudes toward the CPS and CPA portfolios, their responses 
indicate that they saw both as having some value, with mastery of the CPS perceived as 
more important than actually doing a CPA portfolio. 

♦ Having the CPS skills will help me in the future (83% agreeing). 

♦ Employers want to hire people who have the CPS skills (71% agreeing). 

♦ Doing the CPA portfolio will help me in the future (71% agreeing). 

♦ Colleges want to admit students who have them (65% agreeing). 

♦ I recommend that other students do the CPA portfolio. (65% agreeing). 

♦ Having a CPA portfolio will help me in getting a job (64% agreeing). 

♦ The CPA is a good teaching and learning tool (64% agreeing). 

♦ The CPA is a good way to learn and master the CPS skills (62% agreeing). 

The statement related to attitudes toward the CPS/CPA where fewest students were in 
agreement was “Having a CPA portfolio will help me in getting into college,” (54% 
agreed). This is not surprising given the need for additional work to promote and 
understand the CPA as a college admissions tool. 

The attitudes of students from high completion rate and low completion rate schools 
toward the CPS/CPA were compared. The data reported in Table 4 show that students at 
high completion rate schools were more positive about the CPS/CPA than students at 
schools with low completion rates. Additionally, fewer than 60% of students at low 
completion rate schools agreed with 5 of 1 1 statements about the CPA. There was only 
one statement on the Student Survey where students from low completion rate schools 
were more likely to agree with a statement than students from high completion rate 
schools, “Having a CPA portfolio will help me in getting into college” (55% versus 
53%). Given that teachers across the pilot test received the same professional 
development and materials, these data suggest that teacher impact on student support 
for the CPA is substantial. 
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Table 4 



Percent of students who “Strongly Agreed” or “Agreed” with statements related to 
their attitudes toward the CPS/CPA 







High 


Low 


Statement 


PERCENT 


Completion 


Completion 




AGREEING 


Rate 


Rate 




(N=623) 


Schools 


Schools 


Having the CPS skills will help me in the future. 


83% 


88% 


80% 


My teachers thought the CPA portfolio was important. 


76% 


85% 


72% 


Employers want to hire people who have the CPS 
skills. 


71% 


80% 


66% 


Doing the CPA portfolio will help me in the future. 


71% 


79% 


67% 


Colleges want to admit students who have the CPS 
skills. 


65% 


72% 


62% 


1 recommend that other students do the CPA portfolio. 


65% 


75% 


59% 


Having a CPA portfolio will help me in getting a job. 


64% 


71% 


61% 


The CPA is a good teaching and learning tool. 


64% 


74% 


58% 


The CPA is a good way to leam and master the CPS 
skills. 


62% 


74% 


55% 


The CPS are important to my teachers. 


58% 


68% 


53% 


Having a CPA portfolio will help me in getting into 
college. 


54% 


53% 


55% 



Generally positive student attitudes toward the CPA are particularly interesting given that 
student responses about implementation questions suggested some deficiencies. 
Regarding implementation, only 62% of students indicated they understood the CPS, the 
heart of the CPA system, and just 71% reported they understood how to do the entries 
making up the CPA portfolio. Fifty-five percent of students believed their teachers spent 
enough time explaining the CPS and that they received the support and information 
needed to be successful with the CPA. Eight of 1 1 items presented in Table 5, all of 
which positively described the CPA or its implementation, scored below 60% agreement. 

Students from high and low completion rate schools differed substantially in their 
attitudes toward CPS/CPA implementation. Students at high completion rate schools 
were more likely to agree with each positive statement about CPS/CPA implementation. 
At low completion rate schools, fewer than 60% of students agreed with 9 of 1 1 positive 
statements about implementation. These students appear to need much more information 
and support to work with the CPS/CPA. At high completion rate schools, there were 
only four positive statements where agreement was below 60% suggesting these students 
felt more prepared to work with CPA portfolios. Overall, and not surprisingly, these data 
support the hypothesis that teacher impact on student performance is major. 
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Table 5 



Percent of students who “Strongly Agreed” or “Agreed” with statements related to 
CPS/CPA implementation 



Statement 


Percent 

AGREEING 

(N=623) 


High 

Completion 

Rate 

Schools 


Low 

Completion 

Rate 

Schools 


I understood how to do the entries that make up the 


71% 


84% 


64% 


CPA portfolio. 

After doing the CPA portfolio, I have a better ‘ 


67% 


74% 


64% 


understanding of what skills employers and 
colleges want. 

I understood the CPS. 


62% 


74% 


56% 


The CPA “Guidelines for Students” was useful to me. 


56% 


65% 


52% 


My teachers spent enough time explaining the CPS. 


55% 


74% 


44% 


Overall, I received all the information, help, and 


55% 


73% 


45% 


support I needed to be successful with the CPA. 
Doing the CPA portfolio helped me get the skills I 


52% 


61% 


48% 


need to have after I finish high school. 

Using the CPA helps to improve my academic work. 


46% 


49% 


44% 


I can give an accurate assessment of how well I have 


44% 


53% 


40% 


mastered the CPS. 

Teachers who use the CPA teach differently or ask me 


44% 


54% 


38% 


to do different kinds of assignments than teachers 
who do not use the CPA. 

I leam and master the CPS skills better in classes 


38% 


52% 


30% 



where I use the CPA than in classes where I do not 
use the CPA. 



S ummar y of Findings: Students generally found the CPA to be valuable. However, 
survey data suggest that CPA implementation at low performing schools needs more 
teacher time and more focus on the CPS. Less than half the students at low performing 
schools reported that they received the information and help needed to be successful with 
the CPA or that their teachers spent enough time explaining the CPS. 

HOW WOULD STUDENTS IMPROVE THE CPA ? 

Students were asked how they would change the CPA to make it better and what teachers 
could do to help students perform well on the CPA. There were three themes to students’ 
responses. First, a large percentage of students said the CPA needs no changes. They 
like it the way it is. Second, many students indicated that they would like more examples 
that they can use to guide their work. Lastly, students suggested that their teachers take 
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more time to explain the CPA to them. This suggestion is consistent with students’ 
attitudes about CPA implementation. 

WHAT ARE THE KEY CONDITIONS FOR SUCCESSFUL IMPLEMENTATION OF 
THE CPA ? 

Data informing the answer come from the teacher surveys, teacher interviews, student 
survey, and portfolios. 

Teacher Surveys: The teacher surveys explored the question by asking teachers to 
identify what they saw as the five most important factors or resources that actually did 
contribute to the success of the CPA in their classroom during the 1996-97 school year; 
teachers could only identify factors or resources they actually had or used on the 1996-97 
school year. Teachers were provided an extensive list of factors from which they could 
choose. Table 6 presents the percentage of teachers identifying each factor or resource 
they placed in their top five as being most important in their success. 

There were two factors that over 50% of teachers selected: (1) significant numbers of 
students in common with other teachers using the CPA, chosen by 67% of teachers, and 
(2) integrated curriculum, the choice of 53% of teachers. Forty-four percent of teachers 
believed structuring schools into career academies, which generally increases teacher 
collaboration and shared curricular focus, is important to CPA success'*. Together, these 
three factors indicate that teachers believe successful CPA implementation is a shared 
responsibility that requires the coordinated efforts of teachers. 

Professional development, selected by 42% of teachers, was seen as the fourth most 
important factor contributing to successful implementation, followed by the CPA Student 
Guidelines, chosen by 36% of teachers. These two factors point to the importance of 
providing information for teachers to support CPA implementation. In this regard, the 
CPA does not appear to be an assessment which can be implemented “off the shelf’ with 
little orientation about its use, a feature it shares with other performance-based 
assessments and other vehicles intended to support systemic reform. 

Teachers did not widely choose prior experience with portfolios and standards-based or 
performance-based assessments, either on their part or that of their students, as factors 
contributing to successful CPA implementation. These factors were chosen by 19% - 
28% of teachers. This is probably due, at least in part, to the fact that only two of the six 
schools had previous experience with the CPA or in-depth experience with performance- 
based assessment. 



Two career academies participated in the pilot test. 
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Table 6 



Percent of teachers selecting factor as important to success of the CPA in their classrooms 



Factor 


% OF TEACHERS SELECTING 
(N=36) 


Significant numbers of students in common with other 


67% 


teachers using the CPA 


Integrated curriculum 


53% 


Career academies 


44% 


Professional development related to the CPA 


42% 


CPA guidelines 


36% 


More experience with portfolios 


28% 


Students with more experience with portfolios 


25% 


Team teaching 


22% 


Previous experience with standards-based assessment 


19% 


Previous experience with performance-based 


19% 


assessment 


More class time for the CPA 


18% 


Block scheduling 


14% 


More school-wide support of portfolios 


14% 


More experience with the CPS and CPA 


14% 


More school-wide support of Career Preparation 


11% 


Standards 


More school-wide support of the CPA 


6% 


More parent understanding of portfolios 


0% 



The teacher survey also explored the resources or factors that teachers did not have that 
they felt would have made the CPA more successful in their classrooms. Teachers were 
again asked to identify the five most important factors in this area. Their responses are 
provided in Table 7. Number one on teachers’ lists was more experience with the CPS 
and CPA. One of every two teachers believed this would have made CPA 
implementation more successful. Two of the three schools with the highest rates of 
completed and above-Basic portfolios had two years of experience with the CPA. Given 
these results, experience with the CPA might be selected as the most important condition 
for implementation. However, one of the three higher-performing schools did not have 
prior experience with the CPA but did have other conditions identified as key to 
successful implementation including shared student and integrated curriculum. In 
particular, this school demonstrated an extremely strong commitment to the CPA. 

Experience with the CPS/CPA was followed by more professional development related to 
the CPA, more experience with portfolios, and more class time for the CPA. Overall, 
these three factors were chosen by 39% to 42% of teachers. More school-wide support of 
the CPS and the CPA were the next most selected factors with 31% and 25% of teachers 
selecting them, respectively. No other factor was chosen by over 22% of teachers. 
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Table 7 



Percent of teachers selecting factor as one they did not have which would have made the CPA 
more successful in the classroom 



Factor 


Percent of 
Teachers 
Selecting 
(N=36) 


More experience with the CPS and CPA 


50% 


More professional development related to the CPA 


42% 


More experience with portfolios 


39% 


More class time for the CPA 


39% 


More school-wide support of Career Preparation 


31% 


Standards 


More school-wide support of the CPA 


25% 


Students with more experience with portfolios 


22% 


Previous experience with performance-based 


22% 


assessment 


Block scheduling 


22% 


More parent understanding of portfolios 


22% 


Previous experience with standards-based assessment 


19% 


More school-wide support of portfolios 


19% 


Integrated curriculum 


17% 


Career academies 


11% 


Team teaching 


11% 


Different CPA guidelines 


6% 


Different professional development related to the CPA 


4% 


than we received 


Significant numbers of students in common with other 


3% 


teachers using the CPA 



Also informing examination of conditions for successful implementation were questions 
in the teacher survey about how and when they implemented the CPA. These data for 
each of the pilot schools and the pilot study overall are presented in Table 8. Time of 
year when the CPA was introduced and when students began work or entries stand out as 
the factors which most influenced implementation. 

Overall, CPA implementation occurred throughout the school year and teachers 
introduced it from September to April. Over 50% of teachers indicated they introduced 
the CPS to their students by the end of December. Also, the students of over 50% of 
teachers had begun to create their first CPA entries by the end of January. 

However, there were important differences among the pilot schools as to when teachers 
began introducing the CPS and when students completed their first CPA entry . The three 
higher performing schools, now referred to by their pseudonyms, Weishington, Keith, and 
Kennedy, were “early implementers.” Teachers at these schools reported having 
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introduced the CPS by November while lower performing schools often did not begin to 
focus on the CPS until the second semester. At higher performing schools, the first CPA 
entries were begun by December, and students had first drafts of half of their portfolio 
entries written by the end of March. At the low performing schools, this drafting process 
continued until as late as May. 

Table 8 

CPS/CPA Implementation Data 



School 



Characteristic 


Wash 


Tunnel 


Keith 


ROOS. 


Lincoln 


Kennedy 


total 


Number of CPA 
portfolios received 


47 


58 


40 


248 


136 


99 


631 


Percent of received 
CPA portfolios that 
were complete 


49% 


0% 


98% 


2% 


10% 


44% 


20% 


Number of teachers 
surveyed 


5 


6 


3 


11 


7 


4 


36 


Average number of 
teacher’s classes using 
CPA 


4.0 


2.7 


2.0 


2.5 


2.0 


3.8 


3.0 


Average percent of 
teacher’s students 
attempting CPA 


76% 


62% 


40% 


37% 


68% 


100% 


86% 


Average percent of 
teacher’s students 
completing all 
attempted entries 


90% 


100% 


60% 


79% 


93% 


50% 


56% 


Timeline for 
introduction of CPS 


Oct.- 

Nov. 


Oct.- 

March 


Sept. - 
Nov. 


Feb. - 
Apr. 


Sept. - 
March 


Nov. 


Sept. - 
Apr. 


Timeline for first CPA 
entry 


Oct. - 
Dec. 


Nov. - 
Feb. 


Sept. 


Feb. - 
March 


Sept. - 
March 


Nov. - Dec. 


Sept. - 
March 


First drafts of half of 
portfolio entries 


Nov. - 
March 


Feb. - 
Apr. 


Jan. 


Feb. - 
May 


Sept. - Apr. 


Feb. 


Sept. - 
May 


Mean percent of class 
time devoted 


18% 


5% 


20% 


10% 


31% 


7% 


13% 



exclusively to CPA 
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Not surprisingly, the “early implementers” were the schools with the highest percentages 
of completed CPA portfolios (Washington = 49%, Keith = 98%, and Kennedy = 44%) 
and with the highest percentages of portfolios that were Proficient or Advanced. This 
finding suggests that one key to successfully implementing CPA portfolios is to begin 
working with them early in the school year. 

Early implementation may be interpreted as an indicator of teacher commitment to the 
CPA; it is not unreasonable to assume that teachers who believed the CPA and CPS were 
important generally chose to start work on them quite soon in the school year. Certainly, 
WestEd staff who interacted with teachers from all six schools observed that the staff 
from higher performing schools generally appeared more committed to CPA success than 
those at the lower performing schools. Importantly, and supporting the hypothesis that 
teachers at higher-performing schools were more committed to the CPA/CPS, students 
survey responses suggested strongly that teachers at higher performing schools presented 
and taught the CPA and CPS more positively than those at lower performing schools. 
Students at higher performing schools consistently found that the CPS and CPA were 
more valuable and that they had better opportunities to be successful with the CPA than 
those at lower performing schools. 

Other survey topics associated with how and when the CPA was implemented (the 
percent of students attempting the CPA, the percent of students completing all the entries 
they attempted, the average number of classes where the CPA was used, or the percent of 
class time devoted exclusively to the CPA) did not appear to have a significant impact on 
CPA success. 

Interviews with lead teachers: These interviews about key implementation conditions 
elicited four strong themes, teacher commitment, level of program organization and 
communication, experience with the CPA, and professional development. While most 
teachers saw the CPA as valuable, teacher commitment to its success and its importance 
relative to other curricula and assessments varied. Teachers at all three higher- 
performing schools felt the CPA was central to their curricula with one noting the CPA 
“was what we have been looking for.” Schools where teachers had regular opportunities 
to work together also fared better than those that did not. Finally, teachers consistently 
identified experience with the CPA as causal in their success or lack of success. 

Teachers also consistently stated that more or better professional development would not 
have been causal in increasing their success. 
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Summary of Findings: Supportive conditions for successful implementation of the CPA 
appear to be: 

• teacher experience with the CPA and/or portfolios; 

• integrated curriculum; 

• regular and effective communication by implementing teachers; 

• shared students; 

• professional development; 

• initiation of implementation of the CPA early in the school year; and, perhaps most 
importantly, 

• teacher commitment to implementation of the CPA and to the value of the CPS. 

It is worth noting that two of the three higher-performing schools had all of the essential 
conditions with the third having all but one (experience with the CPA and/or portfolios). 
This suggests that successful implementation requires multiple conditions associated with 
whole school change and that schools should be fairly well down the reform path before 
they try it. 
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II. WHOLE SCHOOL CHANGE 



■’ This section of the report seeks to answer the following questions: “Did implementing 

® the CPA cause teachers to change practice or their thinking about practice?” and “Did 

implementing the CPA or embracing the CPS promote whole school change?” The data- 

■ gathering and analyses are informed by the assumptions that causing change in practices 

'* across an entire school takes much more than a year and that one innovation alone (e.g., 

the CPA) is unlikely to cause such change to occur. The primary data source is the 

■ teacher survey. 

I DID IMPLEMENTING THE CPA CA USE TEACHERS TO CHANGE PRACTICE OR 

" THEIR THINKING ABOUT PRACTICE? 

I Responses from the teacher survey suggested the CPS and CPA had modest impacts on 

teaching practices and whole school change (WSC). Regarding “teaching methods”, 

» 58% of teachers “agreed” both that “Using the CPS made me alter or rethink my teaching 

H methods,” and “Using the CPA portfolio as a means of organizing student work made me 

alter or rethink my teaching methods.” Moreover, 53% “agreed” that “After seeing CPA 
H related student work I decided to alter or rethink my teaching methods.” Forty-two 

■ percent “agreed” that “I used the Career Preparation Standards to structure my 

curriculum.” One of these teachers commented, “It brought relevance to the curriculum, 

fl always trying to incorporate CPS into lessons, give examples.” Comments from teachers 

■ who were neutral or disagreed^ about structuring their curriculum in response to the CPA 
indicated they responded that way for the following reasons: 

■ ♦ they had already restructured their curriculum (“I have been using many of the ideas 

of the CPA long before its introduction to our school”); 

■' ♦ their curriculum was fixed for the school year so there was little possibility of 

* restructuring it in response to CPS/CPA implementation (“Since we embarked on the 

^ portfolio project mid-year, curriculum was already set,”); or 

ft ♦ they believed their relation to CPS/CPA implementation was tangential, such as the 

mathematics teacher who commented, “Most of the career preparation standards were 
M discussed in the business class.” This teacher felt there was little reason to change to 

I curriculum. 

V Teachers’ written comments on open-ended survey items often indicated that the 

ft CPS/CPA impacted their teaching. One teacher noted the CPA portfolio provided a 

common focus which increased the ability of academic and vocational teachers to work 
m together. Other comments: 



^ A teacher was considered to “disagree” with a survey item if he/she marked “Disagree” or “Strongly 
Disagree.” 
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♦ “I’m trying to do more with portfolios in my own class.’’ 

♦ “I had to do a lot more on writing skills.’’ 

♦ “I realized students needed much more help in assessing what they’re good at and 
what they’ve accomplished.’’ 

♦ “The need for summaries was an important new tool.’’ 

♦ “I’m emphasking more group work in class to emphasize interpersonal skills...’’ 

♦ “The CPA portfolio supports and is flexible enough to be used in many different 
classroom settings.’’ 

♦ “It helps make school work seem more relevant to the work world.” 

♦ “I want to increase hands-on lab demonstrations with career applications.” 

Regarding students’ perception of teachers who use the CPA and those who do not, it is 
interesting to note that only 44% of students “agreed” that “Teachers who use the CPA 
teach differently or ask me to complete different kinds of assignments than teachers who 
do not use the CPA.” This finding is consistent with the results from the teacher survey 
indicating that the CPA had modest impacts on teaching methods. 

DID IMPLEMENTING THE CPA OR EMBRACING THE CPS PROMOTE WHOLE 
SCHOOL CHANGE? 

Implementation of the CPS/CPA at the pilot schools provided an opportunity to examine 
the impact of standards-based assessment on whole school change, as opposed to 
changing just individual teachers’ methods or curriculum. (Indicators for WSC were: 
development of standards-based assessments and performance-based assessments not 
related to the CPA, community involvement in the setting of standards, use of data 
related to student performance against standards to drive changes in teaching, integrated 
curriculum, block scheduling, team teaching, and career academies.) As Table 9 shows, 
this examination occurred within the context of schools already implementing various 
reforms. Most teachers reported that their schools have been involved with two key 
reform efforts, integrated curriculum (89% of teachers) and setting standards not related 
to the CPS (86% of teachers). Where this standard setting has occurred, 72% of teachers 
indicated that parents, students, employers or community members have been involved in 
the standard-setting process. Career academies, team teaching, and the development of 
standards-based assessments and performance-based assessments not related to the CPA 
were activities occurring at over 55% of the teachers’ schools. 

There were some notable differences between the high and low portfolio completion rate 
schools on these WSC indicators. While use of an integrated curriculum was reported as 
high at both sets of schools, teachers at schools with low completion rates were much 
more likely to report their schools engaged in standard setting, team teaching, and the 
development of standards-based and performance-based assessments. 
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Table 9 



Percent of teachers indicating their school’s involvement with reform activities 



ACTIVITY 


Percent 

AGREEING 

(N=36) 


High 

Completion 

Rate 

Schools 

(N=12) 


Low 

Completion 

Rate 

Schools 

(N=24) 


Integrated Curriculum 


89% 


83% 


92% 


Setting Standards 


86% 


67% 


96% 


Involving parents, students, employers, or community 
members in setting standards 


72% 


75% 


71% 


Career Academies 


69% 


67% 


71% 


Team Teaching 


61% 


25% 


79% 


Developing performance-based assessments 


58% 


25% 


75% 


Developing standards-based assessments 


56% 


25% 


71% 


Block Scheduling 


30% 


33% 


29% 



Overall, many teachers believed that the CPA had caused some WSC and that it is a good 
vehicle to do so. Forty-four percent of teachers indicated their “school or program has 
changed its structure, procedures, or curriculum based on CPA-related student work.” 
Teachers who “agreed” with this statement made the following comments which provide 
some sense of those changes: 

♦ “We now work as teams to coordinate CPA-related work.” 

♦ “We’ve coordinated the various CPA tasks across several classes which has increased 
interdisciplinary work and collegial cooperation.” 

♦ “Our program is integrating more of our curriculum.” 

A large majority of teachers (83%) “agreed” that “The CPA is, or could be, a valuable 
support for implementing standards-based curriculum and assessment school-wide.” A 
few teachers chose to explain their responses to this item. They noted the following: 

♦ “Student analysis of work samples could be a powerful step toward realizing 
goals/standards.” 

♦ “It supports our written, oral, and vocational standards.” 

♦ “For most of the second semester, the entire school worked on the CPA.” 

S ummar y of Findings: Implementing the CPA caused some teachers to change practice 
or their thinking about practice. Also, teachers believe the CPA is a good vehicle to 
affect whole school change. However, the depth to which the CPA caused such change is 
not known - the most successful CPA schools already were implementing components 
of whole school change prior to the pilot test. However, it is important to note that when 
identifying factors that they believe supported successful implementation, teachers often 
identified conditions commonly associated as components of systemic reform. This may 
lead to teachers obtaining those conditions as they attempt to improve CPA 
implementation. 
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III. STUDENT PERFORMANCE/ASSESSMENT 



This section of the report seeks to answer the questions “How well did students perform 
on the CPA?”, “How reliable is the CPA as an assessment?”, and “What is the 
relationship between dimensional and overall scores?” The primary data sources are the 
results of the scoring of 100^ CPA portfolios held in July, 1997 and questionnaires from 
scorers. 

By way of background, WestEd received 63 1 student portfolios from the pilot schools at 
the end of the school year. Of these portfolios, only 20% were “complete”, meaning the 
portfolios contained required entries. The percentage of completed portfolios received 
from individual schools ranged from 98% to 0%. 

Prior to presenting scoring data it is appropriate to describe the scoring process that 
yielded the data. First, a diverse pool of educator and employer scorers were identified 
and recruited. With one exception, all educator scorers came from schools in the pilot 
test. (The one educator not from a pilot test school had substantial experience with the 
CPA and performance-based assessment.) Pilot test teachers were the primary source of 
educator scorers because, based on extensive experience in Kentucky and with the 
Career-Technical Assessment Program, it is likely that teachers involved with the CPA 
will be those who score it. At any given site, it is unlikely that teachers not using the 
CPA will have major responsibilities in scoring CPA portfolios produced at that site. 
Employer scorers all played significant roles in hiring entry-level employees at their 
businesses. Furthermore, they worked at large corporations, small businesses, and 
temporary agencies specializing in entry-level workers. 

Prior to arriving at the two-day scoring session, scorers received detailed information 
about the CPA and two training portfolios for their review. The scoring session began 
with three hours of training focused on close review of seven exemplar portfolios, the 
CPS, and the scoring rubric. Via this training, scorers learned to apply the rubric. 
Following the training, scorers then scored four “calibration” portfolios. Only scorers 
who scored three of the four calibration portfolios correctly had their ratings incorporated 
into the final results. Three of 17 scorers did not calibrate. 

Following calibration in the afternoon of Day One, scorers, who were organized in three 
types of pairings (educator-educator, educator-employer and employer-employer) began 
scoring sets of approximately eight portfolios . In the morning of Day Two, after all 100 
portfolios had been scored twice, additional training was provided to recalibrate the 
scorers to the rubric. Scoring pairs were then reconfigured and the portfolios scored 
again by “new” scorers. 



® No conscious effort was made to score exactly 100 portfolios. This number of scored portfolios was 
cause by adjusting the total set of portfolios to be scored to achieve as equal a representation across schools 
as possible and by the number of available scorers. 



I 

1 

I 

I 

I 

t 

I 

1 

I 

I 

I 

I 

t 

I 

I 



I 

I 

I 




Three levels of performance (Basic, Proficient, and Advanced) were used when giving an 
overall or dimensional (Career Preparation, Analysis, Technology, Communication) 
score. (“Dimensions”, or sub-categories within the rubric that align with the CPS, inform 
the overall rating of a portfolio.) The overall ratings are defined below and the 
dimensional ratings are presented in Table 10. Appendix D contains the entire scoring 
rubric used by the scorers. The overall ratings are: 

BASIC: This rating is given to work that needs improvement and is not ready to 
show an employer. While the work may be complete, it does not show mastery of 
the CPS and does not indicate that the student has the skills necessary for 
satisfactory performance in an entry-level position. 

PROFICIENT This rating means the work is good and is ready to show an 
employer. The work is complete and meets all requirements; it shows mastery of 
the CPS and indicates that the student has the skills necessary for satisfactory 
performance in an entry-level position. 

ADVANCED: This rating means that the work is outstanding. It may exceed the 
expectations of an employer. The work is complete and meets all requirements; it 
shows strong mastery of the CPS and indicate that the student may have the skills 
necessary for exemplary performance in an entry-level position. 

Table 10 



Dimensions for rating the Career Preparation Assessment portfolios 



Dimension 


Features of Dimension 


Career Preparation 


• 

• 


Personal qualities needed for employment 
Interpersonal skills needed for employment 
Career planning and employment literacy 


Analysis 


• 

• 


Analytic thinking 
Evaluation 


Technology 


• 

• 


Presentation of work using technology 
Application of technology other than word 
processing 


Communication 


• 

• 

• 

• 

• 


Attention to audience 

Use of own ideas 

Organization and clarity 

Accuracy and completeness 

Language mechanics, sentence structure, and 

vocabulary 
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To perform reliability analyses and to compare performance across schools and students, 
it was necessary to identify a strategy to give each portfolio one rating which reflects the 
scores of all four scorers (it was unlikely that four scorers would always assign a given 
portfolio the same rating). This combined rating is called “cumulative.” Because the 
Basic-Proficient- Advanced scoring scale is composed of discrete intervals, using a mean 
of the four scores was not appropriate. Instead, a mapping strategy was used to derive 
cumulative scores. 

The mapping strategy was based on the assumption that if at least three of the four 
scorers gave a portfolio the same rating, that rating was appropriate to describe the 
portfolio. In the cases where the four scorers split evenly between two performance 
levels (i.e., when two scorers gave a portfolio a Basic and two gave it a Proficient, or two 
gave it a Proficient and two gave it an Advanced) the authors of this report made 
selective judgments as to how these situations should be resolved. These judgments are 
explained in the following definitions of cumulative categories. Please note the 
introduction of the “Marginally Proficient” category. 

BASIC category — Portfolios were assigned to this category when three or four 
scorers scored the portfolio as “Basic.” 

MARGINALLY PROFICIENT category — Portfolios in this category had been 
scored as Basic by two scorers and as Proficient by two scorers. These portfolios 
appeared to be of a better quality than a Basic portfolio but not as well developed 
as a Proficient portfolio. 

PROFICIENT category — Portfolios were assigned to this category when three or 
four scorers scored the portfolio as “Proficient.” 

ADVANCED category — Portfolios were assigned to this category when at least 
two scorers scored the portfolio as “Advanced.” A decision was made to 
categorize these portfolios in this way since this combination of scores appeared 
infrequently and clearly showed the portfolio was more developed than a 
Proficient portfolio. 

Currently, it is unclear why a portfolio fell into the Marginally Proficient category. The 
work in the portfolio might have been, in fact, at the precise borderline between Basic 
and Proficient. There may be other explanations, however. One explanation is that the 
inconsistencies between the scorers’ scores might reflect variations in how scorers 
applied the scoring rubric. If so, better training may be needed to ensure that scorers 
apply the scoring rubric more consistently. Second, the inconsistencies might have been 
caused by ambiguities in the rubric itself, in which case, the rubric may need 
clarification. Further research will be directed to understand the characteristics of 
Marginally Proficient portfolios and the steps which may need to be taken to allow these 
portfolios to be scored more consistently and to eliminate the category if appropriate. 



HOW WELL DID STUDENTS PERFORM ON THE CPA ? 

Table 1 1 presents data about the scored CPA portfolios. Sixty-two percent of students 
submitting complete portfolios received Proficient or Advanced cumulative scores. 
Nearly one-fourth of the portfolios (23%) were scored Basic. The remaining portfolios 
(15%) were Marginally Proficient. It should be noted that these percentages are not 
representative of the quality of the submitted incomplete portfolios which, while not 
formally scored, were generally of much lower quality than the complete portfolios. 

Table 11 

Percent of CPA portfolios at each performance levef 





Basic 


Marginally 

Prohcient 


Prohcient 


Advanced 


Overall 


23% 


15% 


50% 


12% 


Career Preparation 


11% 


16% 


59% 


14% 


Analysis 


23% 


16% 


55% 


6% 


Technology 


37% 


15% 


33% 


15% 


Communication 


18% 


13% 


57% 


12% 



There was moderate variability in the scoring of portfolios on the CPA dimensions. The 
percent of portfolios judged Proficient or Advanced ranged from a low of 48% in 
Technology to a high of 73% in Career Preparation. The percentage of Marginally 
Proficient portfolios was roughly equal for each dimension, between 13% and 16%, 
similar to the percentage of portfolios judged overall as Marginally Proficient. 

The greater percentage of portfolios judged to be of higher quality in Career Preparation 
is understandable in light of the characteristics of the scoring rubric. The rubric for 
Career Preparation is relatively specific and tightly defined. It speaks in terms of the 
content of student work. The rubric contains references to personal qualities, the ability 
to work with others, and career planning and development. Analysis and Communication 
are broader dimensions and represent higher-order thinking skills. The scoring rubric for 
these two dimensions speaks in terms of skills rather than content. Demonstrating 
Analysis and Communication skills appears to be more difficult than including Career 
Preparation content in portfolios. 

Although the Technology dimension had the highest percentage of portfolios scored as 
Advanced (15%), it also had the highest percentage of portfolios scored Basic (37%). 

The finding that few portfolios contained good or superior quality work in Technology 
seems understandable for two reasons. First, a student must have easy access to a 
computer to build entries for a portfolio rated on Technology as Proficient or Advanced, 
and not all students have easy access. Second, the language of the rubric in the 



’ Percents displayed in Tables 1 1 through 13 may not total 100% due to rounding. 



Technology section was made relatively vague in an attempt to be inclusive in a rapidly 
evolving arena. Likewise, during the training preceding the scoring session, scorers had 
the most difficulty in coming to agreement on what constituted a Basic versus Proficient 
in Technology. 

Portfolio scores arranged by school are presented in Table 12. The school level analysis 
is important because individual schools contained large proportions of given ethnicities 
and/or grade levels. For example, 22 of 23 Washington portfolios came from 10th grade 
students. These represented 71% of all 10th grade portfolios. Keith portfolios were all 
produced by 12th grade students and accounted for 75% of 12th grade portfolios. 
Similarly, 55% of African American students with scored portfolios attended 
Washington, 50% of Latino/Hispanic students attended Lincoln, and 77% of White 
students attended Keith. The uneven distribution of these student characteristics makes it 
difficult to separate the influences of ethnicity or grade level from the affects associated 
with attending a particular school. 

Substantial differences in student performance existed among all the schools and within 
the three higher-performing schools. Re all schools, the highest percent of either 
Proficient or Advanced overall ratings was 83% (Kennedy). The lowest percent of 
Proficient of Advanced overall ratings was 17% (Lincoln). Of the three higher- 
performing schools, Kennedy’s 83% of Proficient or Advanced was significantly greater 
than Washington’s (61%) or Keith’s (63%). No clear reason for the difference in 
performance is readily apparent and it may warrant further investigation. 
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Table 12 



Percent of CPA portfolios at each performance level by pilot school site 



School/Dimension 


Basic 


Marginally 

Proficient 


Proficient 


Advanced 


Keith (N=38) 


Career Preparation 


13% 


13% 


55% 


18% 


Analysis 


18% 


21% 


58% 


3% 


Technology 


8% 


24% 


61% 


8% 


Communication 


45% 


24% 


21% 


11% 


Overall 


16% 


21% 


58% 


5% 


Kennedy (N=23) 


Career Preparation 


4% 


4% 


70% 


22% 


Analysis 


9% 


17% 


57% 


17% 


Technology 


9% 


4% 


44% 


44% 


Communication 


9% 


0% 


57% 


35% 


Overall 


9% 


9% 


48% 


35% 


Lincoln (N=12) 


Career Preparation 


25% 


25% 


50% 


0% 


Analysis 


67% 


8% 


25% 


0% 


Technology 


75% 


17% 


8% 


0% 


Communication 


75% 


8% 


17% 


■ 0% 


Overall 


75% 


8% 


17% 


0% 


Roosevelt (N=4) 


Career Preparation 


0% 


25% 


50% 


25% 


Analysis 


0% 


25% 


75% 


0% 


Technology 


25% 


25%. 


50% 


0% 


Communication 


0% 


25% 


75% 


0% 


Overall 
Tunnel (N=0) 


0% 


25% 50% 

no comolete oortfolios submitted 


25% 


Washington (N=23) 


Career Preparation 


9% 


26% 


61% 


4% 


Analysis 


26% 


9% 


61% 


4% 


Technology 


35% 


9% 


52% 


4% 


Communication 


17% 


9% 


70% 


4% 


Overall 


26% 


13% 


57% . 


4% 



The scores of portfolios are presented by grade level in Table 13.^ Analysis by grade 
level might be confounded by other factors such as school and race because of the 



* Ten students who did not indicate their grade level were not included in the analysis of scores by grade 
level. 
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homogeneity within schools mentioned previously. Eleventh grade students had the 
highest quality portfolios followed by students from the 12th grade, and then the 10th 
grade. Almost all 1 1th grade portfolios were judged overall and on each CPA dimension 
as Proficient or Advanced. This relatively high performance might be attributable to the 
fact that there were only eight 1 1th grade portfolios. The difference between 10th grade 
and 12th grade portfolios was not large, suggesting that the CPA may be implemented 
usefully in the early part of high school. Sixty-two percent of 10th grade and sixty-seven 
percent of 12th grade portfolios were scored overall as Proficient or Advanced. The 
dimension scores of 10th grade students ranged from 59% to 72% Judged Proficient or 
Advanced while the range Judged Proficient or Advanced for 12th grade students was 
42% to 75%. 

Table 13 



Percent of CPA portfolios at each performance level by grade level 



Grade/Dimension 


Basic 


Marginally 

Proficient 


Proficient 


Advanced 


Grade 10 (N=31) 


Career Preparation 


10% 


19% 


65% 


7% 


Analysis 


26% 


13% 


55% 


7% 


Technology 


29% 


13% 


52% 


7% 


Communication 


19% 


10% 


65% 


7% 


Overall 


26% 


13% 


52% 


10% 


Grade 11 (N=8) 


Career Preparation 


0% 


0% 


75% 


25% 


Analysis 


0% 


0% 


75% 


25% 


Technology 


13% 


0% 


50% 


38% 


Communication 


0% 


0% 


63% 


38% 


Overall 


0% 


0% 


63% 


38% 


Grade 12 (N=51) 


Career Preparation 


12% 


14% 


55% 


20% 


Analysis 


16% 


22% 


59% 


4% 


Technology 


39% 


20% 


22% 


20% 


Communication 


8% 


20% 


59% 


14% 


Overall 


16% 


18% 


55% 


12% 



Males and females scores differed with females producing higher quality portfolios than 
males (Table 14). Seventy-one percent of females’ and 50% of males’ portfolios were 
judged Proficient or Advanced. The dimension ratings were closest on Career 
Preparation where 76% of females and 69% of males had a Proficient or Advanced 
portfolio. The gap between females and males in the percent of Proficient or Advanced 
portfolios was much wider for the other CPA diinensions: 27 percentage points for 
Analysis, 26 percentage points for Technology, and 30 percentage points for 
Communication. Chi-squared analyses showed while there were no statistically 
significant differences between females and males overall and on the Technology 
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dimension, females and males differed significantly on Career Preparation, Analysis, and 
Communication. 

Table 14 



Percent of CPA portfolios at each performance level by gender 



Gender/Dimension 


Basic 


Marginally 

PROnCIENT 


PROnCIENT 


Advanced 


Female (N=58) 


Career Preparation 


3% 


21% 


60% 


16% 


Analysis 


10% 


17% 


66% 


7% 


Technology 


31% 


10% 


43% 


16% 


Communication 


9% 


10% 


66% 


16% 


Overall 


16% 


14% 


59% 


12% 


Male (N=42) 


Career Preparation 


21% 


10% 


57% 


12% 


Analysis 


41% 


14% 


41% 


5% 


Technology 


45% 


21% 


19% 


14% 


Communication 


31% 


17% 


45% 


7% 


Overall 


33% 


17% 


38% 


12% 



The scores of African American, Asian/Pacific Islander, Latino/Hispanic, and White 
students are presented in Table 15.^ While a series of chi-square analyses focusing on 
how ethnicity related to student performance found no statistically significant differences 
in their scores overall or on any of the dimensions, analysis by ethnicity is worth noting. 
The percent of Asian/Pacific Islander and White students with Proficient or Advanced 
portfolios was about the same, 70% and 71% respectively. For African American 
students, this figure was 61% while it was 45% for Latino/Hispanic students. The 
percent of students with Proficient or Advanced portfolios was highest for Asian/Pacific 
Islanders on three dimensions; Career Preparation (90%), Analysis (70%), and 
Technology (70%). African American students had the highest percentage of Proficient 
or Advanced portfolios in Communication. Latino/Hispanic students had the lowest 
percentage of 



’ Two students who selected “Other” as their ethnicity and three students who did not indicate their 
ethnicity were not included in the analysis of scores by ethnicity. 
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Table 15 



Percent of CPA portfolios at each performance level by ethnicity 



Ethnic Group/Dimension 


Basic 


Marginally 

PROnCIENT 


PROnCIENT 


Advanced 


African American (N=18) 


Career Preparation 


6% 


28% 


56% 


11% 


Analysis 


22% 


11% 


56% 


6% 


Technology 


28% 


6% 


44% 


22% 


Communication 


11% 


11% 


61% 


17% 


Overall 


17% 


22% 


44% 


17% 


Asian/Paciflc Islander (N=10) 


Career Preparation 


10% 


0% 


60% 


30% 


Analysis 


30% 


0% 


60% 


10% 


Technology 


20% 


10% 


50% 


20% 


Communication 


30% 


0% 


50% 


20% 


Overall 


30% 


0% 


40% 


30% 


Latino/Hispanic (N=20) 


Career Preparation 


10% 


25% 


60% 


5% 


Analysis 


40% 


15% 


40% 


5% 


Technology 


45% 


20% 


25% 


10% 


Communication 


45% 


5% 


45% 


5% 


Overall 


40% 


15% 


40% 


5% 


White (N=47) 


Career Preparation 


13% 


11% 


60% 


17% 


Analysis 


11% 


11% 


60% 


6% 


Technology 


40% 


19% 


26% 


15% 


Communication 


9% 


19% 


60% 


13% 


Overall 


15% 


15% 


60% 


11% 



Proficient or Advanced portfolios on all four dimensions. Their percentage for Career 
Preparation (65%) was just behind that of African American students. The gap between 
Latino/Hispanic and other students was greater for Analysis, Technology, and 
Communication. 



Summary of Findings: Given that this is the first relatively broad pilot test for the CPA, 
it is difficult to state whether student performance was “good,” “bad,” or “mediocre.” 
However, relative to the two earlier pilot tests where no formal scoring was done, student 
performance improved substantially. In particular, Washington and Keith, where 
comparison with the previous pilot tests’ portfolios is possible, it appears their portfolios 
improved a great deal. This supports the hypothesis that experience with the CPA is a 
key factor in success. Perhaps more importantly, the performance of Washington, Keith, 
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and Kennedy students suggests strongly that the CPA is implementable Substantial 
numbers of students from diverse populations can complete CPA portfolios to a 
Proficient, or potentially “hirable,” level. Student performance varied widely across 
gender, race, and school. The cause of variation among race and gender, perhaps 
explained by school, is unclear. 

HOW RELIABLE IS THE CPA AS AN ASSESSMENT? 

The reliability of scores given to CPA portfolios was examined in two ways. First, 
portfolios were identified where: (a) all four scorers agreed on the overall and 
dimensional ratings; (b) three of the four scorers agreed; and (c) two of the scorers 
agreed. Second, an analysis was conducted to determine the degree to which there was 
consistency among pairs of scorers (educator-educator, educator-employer and employer- 
employerX Both of these analyses provided information about how reliably CPA 
portfolios could be scored and whether scoring was consistent across dimensions. 

Four scores: At least three of four scorers gave the same dimension and overall scores 
to portfolios between 70% and 76% of the time. The highest level of agreement was on 
the Analysis dimension (76%), followed by the Communication dimension and the 
Overall rating (each at agreement 75% of the time). Technology (agreement 73% of the 
time), and Career Preparation (agreement 70%, of the time). 

Results, presented in Table 16, showed that there was total agreement among four scorers 
in overall scoring 32% of the time. For each dimension, total agreement occurred on 
Career Preparation for 23% of the portfolios, on Analysis for 29%, on Technology for 
26%, and on Communication for 20% of portfolios. This suggests that the CPA is more 
reliable as an overall assessment of the CPS rather than individual dimensions or CPS. 

Table 16 



Percent of Time Scorers Agree 



Percent 

AGREEMENT 


Career 

Preparation 


Analysis 


Dimension 

Technology 


Communication 


Overall 


Four of Four 












Scorers 


23% 


29% 


26% 


20% 


32% 


At Least Three 












of Four Scorers 


70% 


76% 


73% 


75% 


75% , 



By definition, when four readers rate portfolios on a scale with three categories (Basic, Proficient, and 
Advanced), at least two readers will agree on the rating 100% of the time. 
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Pairs of Scores: Reliability was also measured by examining the percentage of times 
that pairs of scorers agreed in their assessments of the four dimensions and overall 
scores. Three different pairings were made to determine whether being a teacher or an 
employer influenced one’s judgment (educator-educator, educator-employer and 
employer-employer). Since each portfolio was scored four times, there were six scorer 
pairs for each portfolio. Analyses examined how frequently each pair agreed on their 
ratings and how frequently their ratings were in adjacent categories (one rating was Basic 
and the second was Proficient or one rating was Proficient and the other was Advanced). 

Generally, levels of agreement on the overall scores varied little across the three different 
types of pairings (63-65%). Dimensionally, agreement was more likely among teacher 
pairs (agreement ranged from 56-69%, depending on the dimension) than employer pairs 
(agreement ranged from 48-65%, depending on the dimension), with agreement among 
employer-teacher pairs (agreement ranged from 54-64%, depending on the dimension) 
generally in between teachers and employers. There were also differences among teacher 
pairs and employer pairs in where they were most likely to agree. Teacher pairs were 
most likely to agree when scoring the Analysis dimension while employer pairs had 
greatest agreement when scoring Technology. Data about the agreement of scorer pairs 
is presented in Table 17. The differences between the teacher pairs and the employer 
pairs suggest teachers approached the scoring of portfolios with more uniform 
perspectives than employers except in the case of Technology, which interestingly, was 
the dimension least clear to students. 

Table 17 



Percent of time scorer pairs are in agreement 



Dimension 


Pair Type 
Teacher-Teacher 
(N=216 PAIRS) 


Employer- 
Employer 
(N=128 PAIRS) 


Employer-Teacher 
(N=250 PAIRS) 


Career Preparation 


67% 


48% 


54% 


Analysis 


69% 


55% 


60% 


Technology 


56% 


65% 


58% 


Communication 


61% 


58% 


54% 


Overall 


65% 


63% 


64% 



Further evidence relating to the differences between how teachers and employers 
evaluated portfolios was provided by in-depth review of the portfolios scored by both 
teachers and employers. Table 18 shows the percent of teachers and the percent of 
employers scoring these portfolios as Basic, Proficient, and Advanced both overall, and 
on each of the CPA dimensions. Two findings from this analysis are evident. First, 
teachers and employers rated about the same percentage of portfolios as Advanced, 
irrespective of the area under consideration. This finding suggests that scorers had little 
difficulty identifying high quality work. Second, teachers and employers differed in the 
percent of portfolios they rated as Basic or Proficient. Both overall and on each 
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dimension, teachers, compared to employers, rated a greater percentage of portfolios as 
Proficient and fewer as Basic. For Career Preparation, Analysis, and Communication, 
chi-square analyses showed the differences in scores awarded to portfolios by teachers 
and employers were statistically significant (p<.05). For these three dimensions, there 
were apparently clear differences in what teachers and employers viewed as good work 
demonstrating the CPS versus work that did not, even though they were using the same 
scoring rubric. 

Analysis of the scores from the pairs of scorers suggests that when scorers are trained 
particular attention might be focused on increasing teachers’ understanding of the 
Technology dimension and employers’ understanding of the Career Preparation, 
Analysis, and Communication dimensions and the rubric used to score them. These were 
dimensions where less than 60% of teacher-teacher and employer-employer pairs agreed 
on their rating and scorers may have more difficulty making judgments. It is also 
possible that the rubric may need refinements, particularly the distinctions between Basic 
and Proficient. Increasing scorers’ understanding of these dimensions and the scoring 
rubric may increase how uniformly they score, thereby increasing overall reliability. 

Table 18 

A comparison of teachers’ and employers’ ratings of CPA portfolios 



Dimension/role 


Basic 


Proficient 


ADVANCED 


Career Preparation* 


Employer (n=l67) 


29% 


58% 


13% 


Teacher (n=ll6) 


15% 


69% 


16% 


Analysis* 


Employer 


38% 


52% 


10% 


Teacher 


23% 


66% 


11% 


Technology 


Employer 


43% 


43% 


14% 


Teacher 


37% 


46% 


17% 


Communication* 


Employer 


37% 


49% 


14% 


Teacher 


19% 


68% 


13% 


Overall 

Employer 


38% 


52% 


, 11% 


Teacher 


28% 


62% 


10% 



♦Statistically significant difference between Teachers and Employers 



S ummar y of Findings: Given that this is the first attempt at scoring substantial numbers 
of portfolios from a diverse group of schools, the reliability of the CPA was promising. 
Perhaps more importantly, data from this year’s scoring session will inform 
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improvements in the rubric and training techniques for scorers, which should improve 
reliability in subsequent years. Training diverse groups of scorers (e.g., teachers and 
employers) and employers in particular merit increased attention. The finding that 
overall scores had slightly greater reliability than dimensional scores will inform both 
rubric design and the description of the CPA. It may be more appropriate at this time to 
describe the CPA as an assessment of a set of skills rather than as an assessment of 
individual skills. 



WHAT IS THE RELATIONSHIP BETWEEN DIMENSIONAL AND OVERALL SCORES? 

Interest in the relationship between dimensional and overall scores grew out of two 
questions; 1) “Did the scores given to a portfolio’s dimensions differ from its overall 
score?” and 2) “Where dimension scores differed from overall scores, was there a 
dimension which seemed to have greater influence the overall score?” Also, in this 
section, two other questions are examined: “Do scorers prefer a three-point versus a 
four-point scale?” and “Does proficient translate to hirable?” 

Table 19 presents data on how the overall score of a portfolio related to the scores the 
portfolio received on its individual dimensions. The results show that in eight out of ten 
portfolios, 1) the portfolio received the same score (e.g., Basic, Proficient, or Advanced) 
on three, if not all four dimensions, and 2) received that same rating as an overall score. 
In these cases, it appears that the dimensional scorers aligned closely with the overall 
scores. This finding was true for both teachers and employers with no statistically 
significant differences in their response patterns. In 15% of the cases, two portfolio 
dimensions received one score while two other dimensions were scored in the same 
adjacent category, and the portfolio’s overall score was also in one of these two 
categories. Here again, it appears that the dimensional scores aligned closely with the 
overall scores. 

Another 13 portfolios (3%) had at least one dimension that was scored Basic, one scored 
Proficient, and one scored Advanced. These portfolios received an overall rating of 
Proficient. Here, it is difficult to project the role of dimensional scoring on the overall 
score. 

In relatively few cases, the dimension scores did not appear to drive the overall score. 
There were eight portfolios, two percent of all the portfolios rated, where three 
dimensions received one score, the fourth dimension was scored in an adjacent category, 
and the overall rating matched the score given to that fourth dimension. 

To better understand what dimension might dominate overall scoring, portfolios where 
dimensional scores were not in total or strong (three out of four dimensions agreeing) 
agreement were reviewed to see which dimension tended to “win.” It was assumed that 
in the cases where there was not total or strong agreement, the dimensions whose scores 
matched the overall score carried greater weight when generating the overall score. 
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Table 19 



Dimension ratings relation to overall ratings 



Strategy 


Frequency of Use 
(N=398 Scorings) 


Same score given to three or four dimensions 
and the Overall score 


80% of Scorings 


Two dimensions received one score and two 
dimension scored in an adjacent category. 
Overall score matches one of these 
dimensions. 


15% of Scorings 


One dimension rated Basic, one rated 
Proficient, and one rated Advanced with 
Overall score of Proficient 


3% of Scorings 


Three dimension received one score and one 
dimension scored in an adjacent category. 
Overall score matches the one dimension. 


2% of Scorings 



In the cases where two portfolio dimensions received one score while two other 
dimensions were scored in the same adjacent category, and the portfolio’s overall score 
was also in one of these two categories, the Communication dimension received the same 
score as the overall rating 56% of the time. Communication was followed in influence 
by the Analysis dimension, which dominated 47% of cases, and Technology which 
dominated 38% of cases. Career Preparation was the least “influential” dimension. It 
received the same rating as the overall score in only 32% of cases. 

Where three dimensions received one score, the fourth dimension was scored in an 
adjacent category, and the overall rating matched the score given to that fourth 
dimension. Communication again emerged as the dimension which most “influenced” the 
overall score. In four of the eight cases, the score given to Communication matched the 
portfolio’s overall score. Career Preparation and Technology each matched the overall 
score of two portfolios while Analysis was the influential dimension of one portfolio. 

To further investigate which dimension was most influential, a regression analysis was 
conducted to determine how well the average rating that portfolios received on each of 
the CPA dimensions predicted the portfolios’ overall scores. The correlation of each 
dimension with the overall score was high, ranging from .69 for Technology to .93 for 
Communication. The value of for the multiple regression was .91 with 
Communication being the dimension which made the largest contribution to the 
regression equation. The importance of Communication in the regression equation is 



O 

ERIC 



39 



50 



consistent with the finding that Communication was the dimension most closely 
associated with the overall score. 

The final step in examining dimensional influence was a review of the questionnaires 
completed by each of the 14 portfolio scorers (eight CPA teachers and six entry-level 
employers) after the scoring session. The questionnaire asked scorers to make short 
comments about: 

♦ the impact of assigning dimensional scores as they arrived at an overall score; 

♦ which dimensions most influenced their scoring; 

♦ whether the scoring rubric should have three or four score points; and 

♦ whether a “Proficient” score was a good indicator of “hirability”, hirability defined as 
having the skills necessary for satisfactory performance in an entry-level position. 

Regarding the impact of assigning dimensional scores, almost all scorers’ responses 
indicated that they found dimensional scoring helped when formulating an overall score. 
Most scorers commented that the dimensions provided a framework that they used while 
reviewing a portfolio. The dimensions focused their attention and, in some cases, helped 
scorers delineate a student’s strengths and weaknesses. A few scorers wrote that they 
largely ignored dimensional scores when giving portfolios a overall score. They only 
considered the dimensional scores when a portfolio was on the borderline between two 
categories. 

Regarding which dimensions most influenced their scoring, particularly when a portfolio 
appeared to fall between different performance levels, nine of the 14 scorers ranked 
Communication as the most influential dimension. Four other scorers ranked it as the 
second most influential dimension. Each of these four had ranked Career Preparation as 
number one. Comments from some of the scorers put their views of Communication in 
perspective: 



♦ “Communication is the interface through which the student shows him or herself to 
us. If the interface is flawed, it is very difficult to tell the difference between a 
diamond and a piece of glass.” (Educator) 

♦ “The ‘Communication’ factor was most influential based on my personal experience 
that able communicators are more successful in overcoming most problems.” 
(Employer) 

♦ “Good communication skills seems to make the entire portfolio better.” (Educator) 

♦ “If a candidate couldn’t communicate ideas and feelings in writing, it was extremely 
difficult to get any ‘sense’ of the individual. I also think that writing skills are 
indicative of overall preparedness.” (Employer) 

Given that the Communication dimension was identifled as the most influential in 
statistical analyses and by scorers, its influence should be addressed in CPA redesign 
efforts regarding both the appropriate level of influence and its impact on English 
language learners. 
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Three-point versus a four-point scale: Most scorers felt a three-point rubric is 
preferable. These scorers felt the three-point rubric they used worked well, having 
another category would be confusing, and that it is better to have an odd number of 
categories rather than an even number. The three scorers advocating a four-point scale 
argued that there was a wide range in the portfolios judged “Proficient.” They suggested 
that “Highly Proficient” be added to allow scorers to better differentiate among 
“Proficient” portfolios. 

ProHcient and “hirable”: The final issue portfolio scorers considered was whether 
“Proficient” is a good indicator of “hirability.” Most educators felt a “Proficient” 
portfolio indicated a student was ready to move into an entry-level job. Their comments 
included the following: 

♦ “If ‘Proficient’ means that the employer thinks the student will be successfully 
employed at their company with training, then I think it is a good indicator.” 

♦ “Yes, it shows that a student will produce at least satisfactory work.” 

♦ “Yes. I think all the Proficient that I ranked were students with definite potential.” 

Five of the six employers responded to the question about hirability. Four of them felt 
“Proficient” did not necessarily equate to “hirable.” They explained their reasoning in 
the following ways: 

♦ “There are different levels of proficiency.” 

♦ “They may have shown developed skills with great potential but [they] still have a 
way to go before they could be successful on the job.” 

♦ “A student with intent to become an auto mechanic did his portfolio. I ranked his 
product ‘Basic’ but I would have no problems giving him my automobile to ‘fix.’ I 
also feel that he will be very successful.” 

Educators’ and employers’ comments show there is some confusion in two areas. First, 
educators and employers might not understand that a Proficient rating should be a 
valuable indicator suggesting that a student has the skills to succeed in an entry-level 
position versus the erroneous idea that the CPA portfolio is meant to indicate the student 
will be successful in a job. Second, employers and educators might differ on the 
definition of “hirable” at an entry-level position. 

Summary of Findings: Dimensional scores aligned strongly with overall scores. 
Relatively few cases could be identified where dimensional scores did not agree with the 
overall score. The Communication dimension was substantially more likely than the 
other dimensions to align with the overall score when dimensional scores were not in 
total or strong agreement. This suggests that Communication was given greatest weight 
when determining a portfolio’s overall score. Scorers generally felt a three-point scale 
was preferable to a four-point scale. While educators felt Proficient translated well to 
“hirable,” employers often did not. Rubric redesign should address the issue of defining 
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“hirable” and, in particular, how to translate the CPS (a set of skills employers recognize 
as necessary for successful performance in an entry-level position) into descriptors which 
apply across a variety of entry-level jobs. 
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CONCLUSION 



The pilot test shed light on the CPA’s potential as a scalable, reliable assessment that 
supports systemic reform and improved student performance. We now know that the 
CPA can be implemented, at least by a range of diverse schools possessing certain 
preconditions, such that student performance exceeds expectations informed by six years 
of experience with standards-based portfolios (62% of completed portfolios were 
Proficient or better). Furthermore, pilot test teachers believe the CPA is a valuable 
teaching tool that can promote changes in classroom practice and schoolwide reform. 
Conversely, it is clear that successful implementation is in no way assured even when 
implementing schools are stronger in terms of capacity for innovation than the general 
school population -- the three lower-performing CPA schools went through a 
meaningful recruiting and selection process aimed at identifying schools well-positioned 
to use the CPA. Like the three higher-performing schools, the three lower-performing 
schools came from a pool identified as likely to be able to implement innovations well, 
then chose to apply to the pilot test, and then passed a screening test for implementation 
capacity where 75% were rejected. Furthermore, their teachers were committed enough 
to proactively seek out the CPA. Finally, those teachers received supporting professional 
development prior to implementation that they rated as good or better. In spite of all this, 
student performance for these three schools suggested that the CPA may not be 
implemented well even under relatively positive circumstances. We also now know that 
CPA portfolios can be scored with a reasonable level of reliability by a diverse group of 
scorers and that teachers and employers find the performance ratings useful. Moreover, 
we have data about scoring that will almost certainly lead to improved reliability. 

Overall, the CPA has a solid base from which to progress. 

For the CPA to realize its potential and to gain greater understanding of its scalability, it 
must meet interrelated challenges regarding: (1) its reliability as an assessment; (2) 
modification of the portfolio’s entries and scoring rubric to better address the needs of 
employers and college admissions processes; and (3) implementation in a larger number 
of schools representative of the general population of schools with relatively high stakes. 
Regarding reliability and validity, while the CPA needs improvement, it is off to a good 
start technically. Critically, teachers and employers already view a Basic, Proficient, or 
Advanced as more reliable and valid than a letter grade. Now, the CPA needs to achieve 
a level of reliability and validity which allows educators, employers, and parents in 
selected high school communities to use it as a meaningful indicator of student 
performance. This will not be easy given the technical obstacles associated with 
portfolio scoring and external audiences (e.g., parents and policy-makers) that are much 
more accepting of standardized test scores than of cumulative performance-based 
assessments. The following steps should increase reliability: revise the scoring rubric 
with learnings from the 1996-97 pilot test scoring in order to better define the three 
performance levels and dimensions, revise scorer training procedures aimed at building a 
common definition of the three performance levels, and continue to monitor and analyze 
scoring events and results. After taking these steps, reliability should equal or exceed 
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that of the few other available performance-based assessments measuring work readiness 
skills. 

To increase its value as an assessment and its scalability, it must better meet the needs of 
employers and college admissions officers. Regarding employers, the Proficient rating 
must better represent “hirable” in an entry-level position. This is a significant challenge 
as wide variability exists for the definition of hirable among employers of entry-level 
workers. Regarding college admissions, while the CPS align with skills colleges want in 
their students, formal review and validation of the CPS by post-secondary institutions is 
needed. Review of the CPA’s format is also necessary to see if post-secondary 
educators’ concerns about use of portfolios in admissions processes can be addressed. 
Also, it may be appropriate for colleges to pilot the use of the CPA in admissions 
processes to see if the CPA has comparable predictive powers relative to traditional 
admissions indicators such as test scores and grades. Overall, reviews by employers and 
post-secondary admissions officers are likely to lead to changes in the structure of the 
CPA’s entries. 

Implementation in schools more representative of the general population than the pilot 
test schools, and with relatively high stakes (e.g., determining the grade for two or three 
courses or serving as a high school graduation requirement), is essential to better 
understand the CPA’s scalability and impact on whole school change. Implementing the 
CPA in such a fashion would allow deeper understanding of the necessary preconditions 
for successful implementation, particularly with regard to implementation by teachers 
less interested in the CPA than those in the pilot test. It would also allow for increased 
study of external and consequential validity. High stakes might have a variety of 
impacts. They might cause student and/or teacher motivation to increase, perhaps 
decreasing the need for some of the preconditions for success identified in the pilot test. 
Also, high stakes might encourage schools to implement some components of whole 
school change in order to increase performance on the CPA. High stakes might also 
impact the sustainability of the CPA. Additionally, broader use of the CPA would 
provide data for improvements in reliability and scoring procedures in diverse settings. 
High stakes implementation of the CPA in schools more representative of the general 
population than the pilot test schools will require an aggressive recruiting effort and may 
require modification of the CPA and CPS to meet local needs. Willingness to tailor the 
portfolio components and standards to local needs should increase: the pool of schools 
interested in using the CPA as a high stakes assessment, educator desire to use 
performance on the CPA as meaningful data informing systemic change, employer desire 
to use the CPA in hiring decisions, and admissions officers’ desire to use it in admissions 
decisions. 

If the CPA meets the challenges listed above, and if preconditions necessary for 
successful implementation come to exist in a large number of schools, the CPA could 
become a scalable tool that bolsters standards-driven curriculum, performance-based 
assessment, and whole school change. 



APPENDIX A 

CAREER PREPARATION STANDARDS 
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Career Preparation Standards (CPS) 



Personal Skills. Students wiU understand how personal skill development 
affects their employability. They will exhibit positive attitudes, self confidence, 
honesty, perseverance, self-discipline, and personal hygiene. They will manage 
time and balance priorities as well as demonstrate arcapacity for lifelong learning. 

Interpersonal Skills. Students will understand key concepts in group 
dynamics, conflict resolution, and negotiation. They will work cooperatively, share 
responsibilities, accept supervision, and assume leadership roles. They will 
demonstrate cooperative working relationships across gender and cultural groups. 

Thinking and Problem Solving Skills, students wui exhibit critical 

and creative thinking skills, logical reasoning, and problem solving. They will 
apply numerical estimation, measurement, and calculation, as appropriate. They 
will recognize problem situations; identify, locate, and organize needed information 
or data; and propose, evaluate, and select from alternative solutions. 

Communication Skills, students will understand principles of effective 
communication. They will communicate both orally and in writing. They will 
listen attentively and follow instructions, requesting clarification or additional 
information as needed. 

Occupational Safety, students will understand occupational safety issues 
including the avoidance of physical hazards in the work environment. _They will 
operate equipment safely so as not to endanger themselves or others. They will 
demonstrate proper handling of hazardous materials. 

Employment Literacy, students win understand career paths and strategies 
for obtaining employment within their chosen fields. They will assume 
responsibUity for professional growth. They will understand and promote the role 
of their field within a productive society, including the purpose of professional 
organizations. 

Technology Literacy, students will understand and adapt to c hangin g 
technology by identifying, learning, and applying new skills to improve job 
performance. They will effectively employ technologies relevant to their fields. 
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CPA Teacher Survey, 1996-97 

This is a survey about the Career Preparation Assessment (CPA). Because the CPA is new, we are 
collecting information to help guide its development. Your feedback in this survey is critical to the CPA. 
Please answer as honestly and completely as you can. 

ABOUT YOUR SCHOOL AND CLASS(ES) 



1. We are interested in reform efforts at your school. Please check off all activities listed below which 
apply to your school and the approximate number of semesters your school has been engaged in each. 



Activity 
a. 



b. 



c. 



d. 

e. 

f. 

g- 

h. 



Number of Semesters 

Setting Standards (not including 

the Career Preparation Standards 
for Student Performance) 

Involving Parents, Students, 

Employers, or Community 
Members in Setting Standards 

Developing Standards-Based 

Assessments (not including the CPA) 

Performance-based.Assessments ^ 

(not including the CPA) 

Integrated Curriculum 

Block Scheduling • 

Team Teaching 

Career Academies 



2. In what subject(s) did you use the CPA71. 2. 3. 

3. In how many of your classes did you use the CPA? 1 2 3 4 5 6 7 

4. What percentage of your students attempted some part of the CPA? % 

5. What percentage of your students completed all the entries they attempted? % 



6. By what month did you comprehensively introduce the Career Preparation Standards to your students? 

Sept. Oct. Nov. Dec. Jm. Feb. March April May 

7. In what month did your students begin to create their first CPA portfolio entry? 

Sept. Oct. Nov. Dec. Jan. Feb. March April May 

8. In what month had your students completed at least first drafts of half the portfolio entries? 

Sept. Oct. Nov. Dec. Jan. Feb. March April May 

9. On average, what percentage of class time was devoted exclusively to the CPA? % 
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ABOUT THE CPA OVERALL 

The following are a series of statements about the CPA. Please circle the response which best shows how much 
you agree with each statement. 

10. The CPA portfolio is important for my students. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

11. My students were motivated and interested in participating in the CPA. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

12. The Career Preparation Standards represent skills valued by employers. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

13. The Career Preparation Standards represent skills valued by post-secondary educational institutions. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

14. The CPA is a valuable assessment tool for me. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

15. The CPA assesses the Career Preparation Standards well. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

16. Doing the CPA portfolio is a good way for students to learn or master the Career Preparation Standards. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

17. Using the CPA improves students’ academic work. 

.Strongly Agree Agree Neutral Disagree Strongly Disagree 
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18. CPA portfolios are of interest to employers. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

19. CPA portfolios are of interest to post-secondary educational institutions. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

20. I used the Career Preparation Standards to structure my curriculum. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Please explain: 

21. Using the Career Preparation Standards made me alter or rethink my teaching methods. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Please explain: ' 



22. Using the CPA portfolio as a means of organizing student work made me alter or rethink my teaching 
methods. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Please explain: . 



23. After seeing CPA-related student work I decided to alter or rethink my teaching methods. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Please explain: 
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24. Analyzing disaggregated data from the CPA would be valuable. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Please explain; 



25. Our school or program has changed its structure, procedures, or curriculum based on CPA-related 
student work. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Please explain: 



26. The CPA is, or could be, a valuable support for implementing standards-based curriculum and 
assessment schoolwide. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 



Please explain: 



27. I intend to continue using the CPA next year. 

Strongly Agree Agree Neutral Disagree Strongly Disagree 

Please explain: . 



(over) 




28. What factors or resources contributed to the success of the CPA in your classroom? Please check off 
the 5 most important factors. 

a. Integrated curriculum 

b. Previous experience with standards- based assessment 

c. Previous experience with performance-based assessment 

d. Block scheduling 

e. Significant numbers of students in common with other teachers using the CPA 

f. Team teaching 

i. Career academies 

k. Professional development related to the CPA 

l. More class time for the CPA 

m. More experience with the CPS and CPA 

n. CPA guidelines 

o. More experience with portfolios 

p. Students with more experience with portfolios 

q. More parent understanding of portfolios 

r. More schoolwide support of Career Preparation Standards 

s. More schoolwide support of portfolios 

t. More school wide support of the CPA 

u. Other (Please list): 

29. What factors or resources that you did not have this past year would have made the CPA more 
successful in your classroom? Please check off the 5 most important factors. 



a. Integrated curriculum 

b. Previous experience with standards-based assessment 

c. Previous experience with performance-based assessment 

d. Block scheduling 

e. Significant numbers of students in common with other teachers using the CPA 

f. Team teaching 

i. Career academies 

k. More professional development related to the CPA 

l. Different professional development related to the CPA than we received 

m. More class time for the CPA 

n. More experience with the CPS and CPA 

o. Different CPA guidelines 

p. More experience with portfolios 

q. Students with more experience with portfolios 

r. More parent understanding of portfolios 

s. More schoolwide support of Career Preparation Standards 

t More schoolwide support of portfolios 

u. More schoolwide support of the CPA 

V. Other (Please list): ^ ^ 
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30. What changes would you recommend making at your site to support implementation of the CPA? 



31. Outside of the CPA and CPS, are you aware of any activities at your school where standards are being 
developed or used to define what students should know or be able to do? (Please circle your 
response.) 

. Yes No 

If you responded, “Yes,” please describe what those standards are tmd how the CPA relates to them. 



ABOUT PROFESSIONAL DEVELOPMENT FOR THE CPA 



This section is designed to gather information about the professional development you received for the 
Career Preparation Assessment, now that you have worked with it. Please circle the number which best 
shows how much you agree with each of the following statements. 



After the CPA professional development workshops, 
I understood: 


Strongly 


Agree 


Neutral 


Disagree 


Strongly 






Agree 








Disagree 


32. 


the purposes of the CPA portfolio. 


SA 


A 


N 


D 


SD 


33. 


the Career Preparation Standards. 


SA 


A 


N 


D 


SD 


34. 


the components of the CPA. 


SA 


A 


N 


D 


SD 


35. 


the nature of tasks that elicit work demonstrating the CPS. 


SA 


A 


N 


D 


SD 


36. 


how to judge when student work demonstrates the CPS 


SA 


A 


N 


D 


SD 


37. 


how to use the CPA rubric. 


SA 


A 


N 


D 


SD 


38. 


how to design assignments which help students 
demonstrate the CPS. 


SA 


A 


N 


D 


SD 


39. 


how to integrate the CPA into my curriculum plans. 


SA 


A 


N 


D 


SD 


40. 


Please share your comments or suggestions for improving the CPA-related professional development 




your received. 













41. Please share your overall comments or suggestions below for improving the CPA portfolio. 
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CPA Student Survey, 1996-97 

This is a survey about the Career Preparation Assessment (CPA), a portfolio you have been involved in 
this year. Because the CPA is new, we are collecting information to help improve it. Please answer the 
survey questions as honestly and completely as you can. Your feedback is critical to us. Thank you for 
your time and thoughtfulness. 




1. Age: 



2. Grade: (circle one) 



9 10 11 12 



3. Gender: (circle one) 

M F 

4. Ethnicity: (check one) 

1. American Indian 

2. Asian 

3. Pacific Islander 

4. Filipino 

5. Hispanic 

6. Black 

7. White 

8. Other (please describe: ) 

9. Decline to state 

5. N ame, of the class or classes where you worked on the Career Preparation Assessment (CPA) Portfolio: 

a. 

b. 

c. 

d' ^ 

■ . e. 

6. Did you complete all the entries (personal statement, resume, application, letter of recommendation, 
work samples, writing sample, interpersonal skills evaluation) in a CPA portfolio? (Please cirde one.) 




1 



Yes 



No 



I do not know 



BEST COPY AVAILABLE 

66 



(OVER) 



DIRECTIONS: Please circle the number that most correctly corresponds to how much you agree with 
each of the statements in the survey. 







7. I understood the Career Preparation Standards (Thinking and Problem-Solving, Communication, 
Technology Literacy, Personal, Interpersonal, Employment Literacy, and Occupational Safety). 



Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


1 


2 


3 


4 


5 


6 


My teachers spent enough time explaining the Career Preparation 


Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


1 


2 


3 


4 


5 


6 


Employers want to hire people who have the CPS skills. 


Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


1 


2 


3 


4 


5 


6 


Colleges want to admit students who have the CPS skills. 


Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


1 


2 


3 


4 


5 


6 


Having the CPS skills will help 


me in the future. 




Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


1 


2 


3 


4 


5 


6 


The CPS are important to my teachers. 






Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


1 


2 


3 


4 


5 


6 


I can give an accurate assessment of how well I have mastered th( 


Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


1 


2 


3 


4 


5 


6 
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14. 1 understood how to do the entries (resume, work samples, personal statement, etc.) that make up the 
CPA portfolio. 

Strongly Strongly Do Not 

Agree Agree Neutral Disagree Disagr ee _..I^o.w. 

1 "2 3 4 5 6 



15. Overall, I received all the information, help, and support I needed to be successful with the CPA 
portfolio. 

Strongly Strongly Do Not 

Agre e .Agree_ __ Neut ral Disa gree Dis agr e e 

1 2 3 4 5 6 

16. The CPA "Guidelines For Students" was useful to me. 

Strongly Strongly Do Not 

Agree Agree Neu?^l Disagree Disagr ee Know 

f" 2" 3 4 5 6 

17. Doing the CPA portfolio will help me in the future. 

Strongly Strongly Do Not 

Agree Agree ___ Neutral Disagree Disagree Know 

^ ^ - -- 4 5 6 



18. I recommend that other students do the CPA portfolio. 

Strongly Strongly Do Not 

Agree Agr ee Neutral Disagree Disa gree K now 

^ 2 3 4 5 6 




19. After doing the CPA portfolio, I have a better understanding of what skills employers and colleges 
want. 

Strongly Strongly Do Not 

Agree Agree Neutral Disa gree Dis agree 

_ ^ - " 5 6 



I 
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20. Doing the CPA portfolio helped me get the skills I need to have after I finish high school. 



Strongly Strongly Do Not 

Agree Agree Neutral _ Disagree Dis ag ree Know 



21. Having a CPA portfolio will help me in getting a job. 




22 . 



Strongly 



1 

Having a 
Strongly 



1 



Strongly Do Not 

At^ree Neuttal Di sagr e e pisa^ree Know 

*2 3 4 5 6 

CPA portfolio will help me in getting into college. 

Strongly Do Not 

Agree N e u tr al Di sag r ee Disa g r e e Know 

2 3 4 5 6 



23. The CPA is a good way to learn and master the CPS skills. 

Strongly Strongly Do Not 

Agree Agree .Neuoral Di sa g ree Disagree Know 

I ^ ~ - ^ 5.6 

24. Please explain your answer to the previous question. Why did you give the rating (Strongly Agree, 
Agree, Neutral, Disagree, Strongly Disagree) to the last question? 



25. I learn and master the CPS skills better in classes where I use the CPA than in closes where I do 
not use the CPA. 



Strongly 

Ajgree^ 



Agree 

2 



Strongly Do Not 
Ne utral Dis agre e Disagree Know 
3 4 "" 5 ~ 6 ~~ 



26. Please explain your answer to the previous question. Why did you give the rating (Strongly Agree, 
Agree, Neutral, Disagree, Strongly Disagree) to the last question? 
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27. Using the CPA helps to improve my academic work. 



Strongly 

Agree Agree 

....... .. ^... 



Neutral 

" ' "3 



Strongly Do Not 
Disagree Disagree K now 

4 5 6 



28. Teachers who use the CPA teach differently or ask me to do different kinds of assignments 
than teachers who do not use the CPA. 

Strongly Strongly Do Not 

Agree Agree Neutral..... Disagree Disagree I^ow 

29. Please explain your answer to the previous question. Why did you give the rating (Strongly Agree, 
Agree, Neutral, Disagree, Strongly Disagree) to the last question? 



30. My teachers thought the CPA portfolio was important. 



Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


1 


2 


3 


4 


5 


6 


The CPA is a good teaching and learning tool. 




Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 



2 " 3 4 5 6 




32. How would you change the CPA to make it better? 



33. What should teachers do to help students perform well on the CPA? 




Please write on the back side of this page if you need more room. 



THANK YOU 



VERY MUCH!!! 
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EVALUATION 



Career Preparation Assessment Workshop 



We would like to know what support services we may offer that would be 
helpful to you in implementing the CPA. Please check all that apply and 
indicate whether you feel the service would be; a) not essential, b) helpful, 
or c) critical, to your success. 

site visits to your school and individual classes 

local standards development (tailored CPS) 

local rubric development 

presentations to parents, local employers, and/or community 

members 

scoring workshops 

The quality of the facilitators/presenters was: 

O Excellent O Good O Fair O Poor 
The quality of the meeting materials were: 



O Excellent O Good O Fair O Poor 
In terms of personal enjoyment, I found the meeting: 



□ Excellent □ Good □ Fair □ Poor ■ 

As a use of my time, the meeting was: 

□ Excellent □ Good □ Fair □ Poor 

The meeting achieved Goal 1 : 

□ Very Well □ Well □ Somewhat □ Not at all 
The meeting achieved Goal 2: 



□ Very Well □ Well □ Somewhat 



□ Not at all 



The meeting achieved Goal 3: 

□ Very Well □ Well □ Somewhat □ Not at all 



□ Somewhat 



□ Not at all 



9. The meeting achieved Goal 4: 

□ Very Well □ Well 

10. The meeting achieved Goal 5: 

□ Very Well □ Well □ Somewhat □ Not at all 

1 1 . The meeting achieved Goal 6: 

□ Very Well □ Well □ Somewhat □ Not at all 

12. The meeting achieved Goal 7: 

□ Very Well □ Well □ Somewhat □ Not at all 

1 1 . Overall, I would describe attainment of the meeting goals as: 

G Very Well G Well G Somewhat G Not at all 

12. What in the meeting could have been better/improved? 



13. What in the meeting was particularly useful to you? 



1 4. Other Comments: 
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CPA Evaluation Workshop #2 
Evaluation Form 



Agree=1 Somewhat Agree=2 No Opinion = 3 Somewhat Disagree=4 Disagree=5 

1 . The goals for the meeting were appropriate. 

Comments 



2. Goal #1 was accomplished. 
Comments 



3. Goal #2 was accomplished. 
Comments 



4. Goal #3 was accomplished. 
Comments 



5. Goal #4 was accomplished. 

Comments 



6. Facilitator was knowledgeable and helpful. 

Comments 



7. Sufficient time was allowed for large/small group work. 
Comments 



8. My questions were answered satisfactorily. 
Comments 



9. This meeting was a good use of my time. 
Comments 



10. What I liked most: 



11. What I liked least: 



12. Suggestions for next time or general comments: 



CPA Evaluation Workshop #3 
Evaluation Form 



Agree=1 Somewhat Agree=2 No Opinion = 3 Somewhat Disagree=4 Disagree=5 

1 . The goals for the meeting were appropriate. 

Comments ^ 



2. Goal #1 was accomplished. 
Comments 



3. Goal #2 was accomplished. 
Comments 



4. Goal #3 was accomplished. 
Comments 



5. Goal #4 was accomplished. 
Comments 



6. Facilitator was knowledgeable and helpful. 
Comments 



7. Sufficient time was allowed for large/small group work. 
Comments 



8. My questions were answered satisfactorily. 
Comments 



9. This meeting was a good use of my time. 
Comments 



10. What I liked most: 



1 1 . What I liked least: 



12. Suggestions for next time or general comments: 
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CPA Scorer Survey 



Name: 



Educator or Employer 



Did dimensional scoring: 1 help 2 hinder 3 not have an impact on 4 other 
arriving at a holistic score? Please circle one of the four choices and write at least two 
sentences explaining your answer. 



Please rank order the dimensions in terms of influence over your scoring when a portfolio 
wavered between different performance levels. “1” is the most influential. Please write 
at least two sentences explaining your answer. 

Career Preparation 

Analysis 

Technology 

Communication 



Should the rubric have 4 score points? 1 Yes 2 No 3 Other. Please circle one of the 
3 choices and write at least two sentences explaining your answer. 



Is Proficient a good indicator of “hireability”? Yes, two more sentences please. 
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APPENDIX C 



Results from 

CPA TEACHER SURVEY 
CPA STUDENT SURVEY 



Spring, 1997 



CPA Teacher Survey, 1996-97 

This is a survey about the Career Preparation Assessment (CPA). Because the CPA is new, we are 
collecting information to help guide its development. Your feedback in this survey is critical to the CPA. 
Please answer as honestly and completely as you can. 

ABOUT YOUR SCHOOL AND CLASS(ES) 

1. We are interested in reform efforts at your school. Please check off all activities listed below which 
apply to your school and the approximate number of semesters your school has been engaged in each. 



a. 


Activity 

86% 


Setting Standards (not including 


b. 


72% 


the Career Preparation Standards 
for Student Performance) 
Involving Parents, Students, 


c. 


56% 


Employers, or Community 
Members in Setting Standards 
Developing Standards-Based 


d. 


58% 


Assessments (not including the CPA) 
Performance-based Assessments 


e. 


89% 


(not including the CPA) 
Integrated Curriculum 


f. 


31% 


Block Scheduling 


g- 


61% 


Team Teaching 


h. 


69% 


Career Academies 



Number of Semesters 



2. In what subject(s) did you use the CPA?1. 2. 3. 

3. In how many of your classes did you use the CPA? 

1 2 3 4 5 6 7 No Response 

11% 22% 36% 17% 11% 0% 0% 3% 



4. What percentage of your students attempted some part of the CPA? 

5. What percentage of your students completed all the entries they attempted? 



.% 



.% 



6. By what month did you comprehensively introduce the Career Preparation Standards to your students? 



Sept. 

6% 



Oct. 

11% 



Nov. 

28% 



Dec 

11% 



Jan. 

0% 



Feb 

31% 



March 

6% 



April 

6% 



7. In what month did your students begin to create their first CPA portfolio entry? 
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Sept. 

8 % 



Oct. 

6% 



Nov. 

22 % 



Dec 

8 % 



Jan. 

11 % 



Feb 

25% 

77 



March 

14% 



April 

3% 



May 

0% 



May 

0% 



No Response 
3% 



No Response 

3% 



(over) 



8. In what month had your students completed at least first drafts of half the portfolio entries? 

Sept. Oct. Nov. Dec Jan. Feb March April May No Response 
3% 0 % 3 % 3 % 6 % 28 % 17 % 25 % 8 % 8 % 



9. On average, what percentage of class time was devoted exclusively to the CPA? % 

ABOUT THE CPA OVERALL 

The following are a series of statements about the CPA. Please circle the response which best shows how much 
you agree with each statement. 

10. The CPA portfolio is important for my students. 

Strongly Agree Agree Neutral Disagree Strongly Disagree No Response 

36% 58% 3% 0% 0% 3% 



1 1. My students were motivated and interested in participating in the CPA. 

Strongly Agree Agree Neutral Disagree Strongly Disagree No Response 

8% 64% 17% 8% 0% 3% 



12. The Career Preparation Standards represent skills valued by employers. 

Strongly Agree Agree Neutral Disagree Strongly Disagree No Response 

53% 36% 6% 0% 0% 6% 



13. The Career Preparation Standards represent skills valued by post-secondary educational institutions. 



Strongly Agree Agree 

19% 64% 



Neutral 

8 % 



Disagree 

3% 



Strongly Disagree 
0% 



No Response 
6% 



14. The CPA is a valuable assessment tool for me. 



Strongly Agree Agree 

11% 64% 



Neutral Disagree Strongly Disagree No Response 
17% 6% 0% 3% 



15. The CPA assesses the Career Preparation Standards well. 



Strongly Agree Agree 

36% 44% 



Neutral 

17% 



Disagree 

0% 



Strongly Disagree 
0% 



No Response 
3% 



16. Doing the CPA portfolio is a good way for students to learn or master the Career Preparation Standards. 



Strongly Agree 
36% 




Agree 

50% 



Neutral 

11% 



Disagree Strongly Disagree No Response 
0% 0% 3% 
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17. Using the CPA improves students’ academic work. 



Strongly Agree Agree Neutral 

17% 58% 19% 

18. CPA portfolios are of interest to employers. 

Strongly Agree Agree Neutral 

22% 56% 22% 

19. CPA portfolios are of interest to post-secondary 

Strongly Agree Agree Neutral 

14% 36% 33% 



Disagree 

3% 


Strongly Disagree 
0% 


No Response 
3% 


Disagree 

0% 


Strongly Disagree 
0% 


No Response 
0% 


educational institutions. 




Disagree 

14% 


Strongly Disagree 
0% 


No Response 
3% 



20. 1 used the Career Preparation Standards to structure my curriculum. 

Strongly Agree Agree Neutral Disagree Strongly Disagree No Response 

3% 39% 33% 19% 3% 3% 

Please explain: 



21. Using the Career Preparation Standards made me alter or rethink my teaching methods. 

Strongly Agree Agree Neutral Disagree Strongly Disagree No Response 

6% 53% 33% 3% 3% 3% 

Please explain: 



22. Using the CPA portfolio as a means of organizing student work made me alter or rethink my teaching 
methods. 

Strongly Agree Agree Neutral Disagree Strongly Disagree No Response 

3% 56% 31% 0% 3% 8% 

Please explain: ^ ^ 
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23. After seeing CPA-related student work I decided to alter or rethink my teaching methods. 

Strongly Agree Agree Neutral Disagree Strongly Disagree No Response 

6% 47% 39% 6% 3% 0% 

Please explain: 



24. Analyzing disaggregated data from the CPA would be valuable. 

Strongly Agree Agree Neutral Disagree Strongly Disagree No Response 

3% 31% 36% 7% 0% 25% 

Please explain: 



25. Our school or program has changed its structure, procedures, or curriculum based on CPA-related 
student work. 

Stron^y Agree Agree Neutral Disagree Strongly Disagree No Response 

8% 36% 25% 22% 6%' 3% 

Please explain: 



26. The CPA is, or could be, a valuable support for implementing standards-based curriculum and 
assessment schoolwide. 

Strongly Agree Agree Neutral Disagree Strongly Disagree No Response 

33% 50% 17% 0% 0% 0% 

Please explain: 
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27. I intend to continue using the CPA next year. 



Strongly Agree Agree 

36% 56% 



Neutral 

3% 



Disagree 

0% 



Strongly Disagree 
3% 



No Response 
3% 



Please explain: 



28. What factors or resources contributed to the success of the CPA in your classroom? Please check off 
the 5 most important factors. 

Integrated curricuium 

Previous experience with standards-based assessment 
Previous experience with performance-based assessment 
Block scheduling 

Significant numbers of students in common with other teachers using the CPA 
Team teaching 
Career academies 

Professional development related to the CPA 
More class time for the CPA 
More experience with the CPS and CPA 
CPA guidelines 

More experience with portfolios 
Students with more experience with portfolios 
More parent understanding of portfolios 
More schoolwide support of Career Preparation Standards 
More school wide support of portfolios 
More schoolwide support of the CPA 

Other (Please list): 

29. What factors or resources that you did not have this past year would have made the CPA more 
successful in your classroom? Please check off the 5 most important factors. 



a. 


53% 


b. 


19% 


c. 


19% 


d. 


14% 


e. 


67% 


f. 


22% 


i. 


44% 


k. 


42% 


1. 


17% 


m. 


14% 


n. 


36% 


0. 


28% 


P- 


25% 


q- 


0 % 


r. 


11% 


s. 


14% 


t 




u. 
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a. 


17% 


Integrated curriculum 


b. 


19% 


Previous experience with standards-based assessment 


c. 


22% 


Previous experience with performance-based assessment 


d. 


22% 


Block scheduling 


e. 


2% 


Significant numbers of students in common with other teachers 


f. 


11% 


Team teaching 


i. 


11% 


Career academies 


k. 


42% 


More professional development related to the CPA 


1. 


11% 


Different professional development related to the CPA than we 


m. 


39% 


More class time for the CPA 


n. 


50% 


More experience with the CPS and CPA 


0. 




Different CPA guidelines 


P- 


39% 


More experience with portfolios 


q- 


22% 


Students with more experience with (X)itfolios 


r. 


22% 


More parent understanding of portfolios 


s. 


2i% 


More school wide support of Career Preparation Standards 


t. 


19% 


More schoolwide support of portfolios 


u. 


25% 


More school wide support of the CPA 


V. 




Other ^Please list): 
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30. What changes would you recommend making at your site to support implementation of the CPA? 



31. Outside of the CPA and CPS, are you aware of any activities at your school where standards are being 
developed or used to define what students should know or be able to do? (Please circle your 
response.) 

Yes 61% No 39% 

If you responded, “Yes,” please describe what those standards are and how the CPA relates to them. 



1 



er|c 
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ABOUT PROFESSIONAL DEVELOPMENT FOR THE CPA 



This section is designed to gather information about the professional development you received for the 
Career Preparation Assessment, now that you have worked with it. Please circle the number which best 
shows how much you agree with each of the following statements. 

After the CPA professional development workshops, 



I understood: 


Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


No 

Responsf 


32. 


the purposes of the CPA portfolio. 


47% 


42% 


3% 


0% 


0 % 


8% 


33. 


the Career Preparation Standards. 


36% 


47% 


8% 


0% 


0% 


8% 


34. 


the components of the CPA. 


44% 


39% 


8% 


0% 


0% 


8% 


35. 


the nature of tasks that elicit work demonstrating the CPS. 


14% 


53% 


19% 


6% 


0% 


8% 


36. 


how to judge when student work demonstrates the CPS 


8% 


61% 


17% 


6% 


0% 


8% 


37. 


how to use the CPA rubric. 


17% 


50% 


22% 


3% 


0% 


8% 


38. 


how to design assignments which help students 
demonstrate the CPS. 


11% 


28% 


42% 


8% 


0% 


11% 


39. 


how to integrate the CPA into my curriculum plans. 


17% 


25% 


39% 


6% 


3% 


11% 



40. Please share your comments or suggestions for improving the CPA-related professional development 
your received. 



41. Please share your overall comments or suggestions below for improving the CPA portfolio. 




Thank you for your feedback 



7 
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CPA Student Survey, 1996-97 

This is a survey about the Career Preparation Assessment (CPA), a portfolio you have been involved in 
this year. Because the CPA is new, we are collecting information to help improve it. Please answer the 
survey questions as honestly and completely as you can. Your feedback is critical to us. Thank you for 
your time and thoughtfulness. 




1. Age; 



2. Grade: (circle one) 



9 10 11 12 No Response 

22% 25% 28% 23% 2% 

3. Gender: (circle one) 

M F No Response 

42% 57% 1% 



4. Ethnicity: (check one) 



1% American Indian 
2% Asian 
1% Pacific Islander 
4% Filipino 
34% Hispanic 
26% Black 
14% White 

9% Other (please describe; ) 

8% Decline to state 



5. Name of the class or classes where you worked on the Career Preparation Assessment (CPA) 
Portfolio; 

e. 

f. 

g- 

h. 

i. 
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6. Did you complete all the entries (personal statement, resume, application, letter of recommendation, 
work samples, writing sample, interpersonal skills evaluation) in a CPA portfolio? (Please circle one.) 

Yes I do not know No Response 

46% 37% 17% 1% 



DIRECTIONS: Please circle the number that most correctly corresponds to how much you agree with 
each of the statements in the survey. 






/looter 




7. I understood the Career Preparation Standards (Thinking and Problem-Solving, Communication, 
Technology Literacy, Personal, Interpersonal, Employment Literacy, and Occupational Safety). 

Strongly Strongly Do Not No 

Agree Agr ee l^cutrd Disagree Disa gree Know Res ponse 

™ 

8. My teachers spent enough time explaining the Career Preparation Standards (CPS). 

Strongly Strongly Do Not No 

_Agree Agree_ _Neutr^ Disagr ee Disagree Know Resp onse 

~ j~~ ~Y6% 72% 6% 2% 7% 



9. Employers want to hire people who have the CPS skills. 

Strongly Strongly Do Not No 

__Agree A gree Ne utral Disagree Disagree Know Response 





33% 


38% . 


16% 2% 


2% 


6% 


3% 


10. 


Colleges want to admit students who have the CPS skills. 






Strongly 

Agree 


Agree 


Neutral Disagree 


Strongly 

Disagree 


Do Not 
Know 


No 

Response 


















29% 


37% 


21% 4% 


2% 


6% 


1% 


11. 


Having the CPS skills will help me in the future. 








Strongly 

Agree 


Agree 


Neutral Disagree 


Strongly 

Disagree 


Do Not 
Know 


No 

Response 



42% 40% 11% 2% 1% 3% 1% 
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12. The CPS are important to my teachers. 

Strongly Strongly Do Not No 

Agree Agrw Neutral Disagree Disagree Know Resj)onse 



22% 36% 27% 6% 2% 7% 1% 

13. I can give an accurate assessment of how well I have mastered the CPS. 



Strongly Strongly 

Agree _ Agree Neutral Disagree Disagree 



Do Not 
Know 



No 

Response 



1 



5 



37 



35 



10 
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14. 1 understood how to do the entries (resume, work samples, personal statement, etc.) that make up the 
CPA portfolio. 



Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


No 

Response 


1 


2 


3 


4 


5 


6 




27 


44 


19 


6 


1 


3 


1% 


15. Overall, I received all the information, help, and support I needed to be suco 
portfolio. 


Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


No 

Response 


1 


2 


3 


4 


5 


6 




17 


38 


27 


13 


3 


2 


1% 


16. The CPA 


"Guidelines For Students" was useful to me. 






Strongly 


- Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


No 

Response 


1 


2 


3 


4 


5 


6 




15 


42 


29 


6 


3 


5 


i% 


17. Doing the CPA portfolio will help me in the future. 






Strongly 
Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


No 

Response 




2 


3 


4 


5 


6 




26 


45 


18 


4 


1 


4 


1% 



O 
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18. I recommend that other students do the CPA portfolio. 



Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


No 

Response 


1 


2 


3 


4 


5 


6 






39 


21 


7 


3 


3 


1% 



ABoUf VAUU6 OP- yooA, CfA 




1 Q After doing the CPA portfolio. I have a better understanding of what skills employers and colleges 


want. 










Strongly 




Strongly 


Do Not 


No 


Agree Agree 


Neutral Disagree 


Disagree 


Know 


Response 


1 2 


3 4 


5 


6 




24 43 


22 6 


1 


3 


1% 


20. Doing the CPA portfolio helped me get the skills I need to have after I finish high school. 


Strongly 




Strongly 


Do Not 


No 


Agree Agree 


Neutral Disagree 


Disagree 


Know 


Response 


1 2 


3 4 


5 


6 




12 40 


31 9 


3 


3 


3% 


21. Having a CPA portfolio will help me in getting a job. 





Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


No 

Response 




1 


2 


3 


4 


5 


6 






19 


45 


23 


j 


1 


3 


3% 




22. Having a 


CPA portfolio will help me in getting into college. 






Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


No 

Response 




1 


2 


3 


4 


5 


6 






14 


40 


29 


6 


2 


6 


3% 




23. The CPA is a good way to leam and master the CPS skills. 






Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


No 

Response 




1 


2 


3 


4 


5 


6 






16 


45 


23 


4 


1 


8 


3% 
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24. Please explain your answer to the previous question. Why did you give the rating (Strongly Agree, 
Agree, Neutral, Disagree, Strongly Disagree) to the last question? 



25. I learn and master the CPS skills better in classes where I use the CPA than in classes where I do 
not use the CPA. 

Strongly Strongly Do Not No 

Ajree A gree Neutral Disagree JDisagree Know _ Re^onse 

..™.... _ ^ ^ ^ „ 

.„ 

26. Please explain your answer to the previous question. Why did you give the rating (Strongly Agree, 
Agree, Neutral, Disagree, Strongly Disagree) to the last question? 



27. Using the CPA helps to improve my academic work. 



Strongly Strongly Do Not No 

Agree Agr e e Neutral Disagree Disagree Kno w Response 

1 ' '2'" 3 4 J ^ 6^ 

8 38 34 8 2 's 5% 



28. Teachers who use the CPA teach differently or ask me to do different kinds of assignments 
than teachers who do not use the CPA. 

Strongly Strongly Do Not No 

Ag ree _ A gree Neutral Disagree Disagree K now Respon se 

™ Q 3 4_" 6 ” 

”7o 34 25 11 ~ ' 1 ~l2 6% 

29. Please explain your answer to the previous question. Why did you give the rating (Strongly Agree, 
Agree, Neutral, Disagree, Strongly Disagree) to the last question? 
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30. My teachers thought the CPA portfolio was important. 



Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


No 

Response 


1 


2 


3 


4 


5 


6 


... - 


36 


40 


id 


2 


I 


5 


6 


The CPA is a good teaching and learning tool. 






Strongly 

Agree 


Agree 


Neutral 


Disagree 


Strongly 

Disagree 


Do Not 
Know 


No 

Response 


1 


2 


3 


4 


5 


6 




19 


45 


20 


4 


2 


4 


6 




32. How would you change the CPA to make it better? 



33. What should teachers do to help students perform well on the CPA? 



Please write on the back side of this page if you need more room. 
THANK YOU VERY MUCH!!!!! 



O 
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APPENDIX D 



CAREER PREPARATION ASSESSMENT RATING GUIDE 



o 



90 



I 



I 

I 

I 

I 

I 

I 

I 

D 

I 




0 

“D 



0 

O) 



Oj 

CL 

< 

CL 

o 



0 

E 

0 

0 

0 

0 

0 

< 

C 

o 

0 

0 

CL 

0 



0 

0 

0 

o 



COMMUNICATION 

• Allention to audience 

• Using own ideas 

• Organization and clarity 

• Accuracy and completeness 

• Language mechanics, sentence 
structure, and vocabulary 


• Shows little or no awareness of 
the audience 

• Writing is not original; copies 
the ideas of others 

• Ideas are presented in a 
disorganized way 

• Work lacks, accuracy and 
completeness 

• Writing contains errors in 
language use that make ideas 
difficult to understand 


• Effectively presents self and 
ideas to outside reviewer 

• Writing is original 

• Writing is clear and organized 

• Work is accurate and fully 
developed 

• Writing contains minor language 
errors; ideas are 
understandable 


• Self and ideas “come alive" to 
outside reviewer 

• Writing is original and may be 
creative 

• Writing is clear and well 
organized throughout portfolio 

• Work is accurate and complete 
With consistent and superior ; 
development; shows attention lo 
detail 

• Writing is almost free of 
language errors and is easy to 
understand 


TECHNOLOGY 

• Presentation of work using 
technology 

• Application of technology 
other than word processing 

t 


• Does not use technology to 
present work (Appearance 
interferes with presentation of 
work) 

• Application of technology other 
than word processing is 
ineffective or lacking 


• Uses technology to present 
work 

• Effectively applies technology 
other than word processing 


• :^Uses technology to enhance 
v|presentation of work 
■!* Ivfcffectively applies technology 
f|i;‘||6ther than word processing 
ll'that is relevant to chosen field 


ANALYSIS 

• Analytic thinking 

• Evaluation 


• Reasoning is unclear, illogical, 
or superficial; interprets or 
calculates information 
inaccurately; makes 
statements with little 
explanation . 

• Gives incomplete or sketchy 
evaluation of own work 


• Reasoning is clear and logical; 
interprets or calculates 
information accurately; 
supports statements with 
explanation 

• Gives accurate evaluation of 
own work 


• Reasoning is clear, logical, and 
thorough; interprets or . 

V.calculates information v r 
'accurately aiid creativelyXXX. 
Supports statemehts with' ' 

■ evidence 

• Shows understanding and 

X); Insight in evaluating own work 

K.'x' X X ■ 

’■i'XXx X;.X' ,'ri -X'- ' . . X'’'' 


CAREER PREPARATION 

• Personal i|ualities needed 
lor employment 

• Interpersonal skills needed 
(or employment 

• Career planning and 
employment literacy 


• Does not identify own 
personal qualities needed to 
be successfully employed 

• Shows little or no ability to 
work productively with 
others 

• Shows little evidence of 
planning for a career 


• Identifies own personal 
qualities needed to be 
successfully employed 

• Shows ability to work 
productively with others 

• Shows evidence of planning 
and developing a career 


• Consistently highlights own 
personal qualities needed to 
be successfully employed 

• Shows leadership and strong 
ability to v/ork productively 
with others 

• Shows excellent 
understanding of career 
planning; describes realistic 
plan (or achieving carepr 




BASIC 

(Not ready 
to show 
employer or 
college) 


PROFICIENT 

(Ready 
to show 
employer or 
college) 


ADVANCED 

(Superior 
quality, may 
exceed 
expectations 
of employer 
or college) 




PA Portfolio Guidelines 



CV 

05 



-fH 

05 



41 



bestcopyavaii^^e 




U.S. Department of Education 

Office of Educational Research and Improvement (OERI) 
National Library of Education (NLE) 
Educational Resources Information Center (ERIC) 




NOTICE 



Reproduction Basis 




This document is covered by a signed "Reproduction Release 
(Blanket)" form (on file within the ERIC system), encompassing all 
or classes of documents from its source organization and, therefore, 
does not require a "Specific Document" Release form. 




This document is Federally-funded, or carries its own permission to 
reproduce, or is otherwise in the public domain and, therefore, may 
be reproduced by ERIC without a signed Reproduction Release form 
(either "Specific Document" or "Blanket"). 




EFF-089 (3/2000) 




